# embedding_metadata

key: chroma:document
string_value:
2. Minimize memory fragmentation with it's smart GPU memory management system which also allows you to fit bigger models and data batches.

<Tip>

Set the `allgather_bucket_size` and `reduce_bucket_size` values to 2e8 in the [ZeRO-2](#zero-configuration) configuration file to get better performance on a single GPU.

</Tip>

</hfoption>
</hfoptions>

### Multi-node deployment

A node is one or more GPUs for running a workload. A more powerful setup is a multi-node setup which can be launched with the `deepspeed` launcher. For this guide, let's assume there are two nodes with 8 GPUs each. The first node can be accessed `ssh hostname1` and the second node with `ssh hostname2`. Both nodes must be able to communicate with each other locally over ssh without a password.

By default, DeepSpeed expects your multi-node environment to use a shared storage. If this is not the case and each node can only see the local filesystem, you need to adjust the config file to include a [`checkpoint`](https://www.deepspeed.ai/docs/config-json/#checkpoint-options) to allow loading without access to a shared filesystem:

```yaml
{
  "checkpoint": {
    "use_node_local_storage": true
  }
}
```

You could also use the [`Trainer`]'s `--save_on_each_node` argument to automatically add the above `checkpoint` to your config.

<hfoptions id="multinode">
<hfoption id="torchrun">

For [torchrun](https://pytorch.org/docs/stable/elastic/run.html), you have to ssh to each node and run the following command on both of them. The launcher waits until both nodes are synchronized before launching the training.

```bash
torchrun --nproc_per_node=8 --nnode=2 --node_rank=0 --master_addr=hostname1 \
--master_port=9901 your_program.py <normal cl args> --deepspeed ds_config.json
```

</hfoption>
<hfoption id="deepspeed">

For the `deepspeed` launcher, start by creating a `hostfile`.

```bash
hostname1 slots=8
hostname2 slots=8
```

Then you can launch the training with the following command. The `deepspeed` launcher automatically launches the command on both nodes at once.

```bash
deepspeed --num_gpus 8 --num_nodes 2 --hostfile hostfile --master_addr hostname1 --master_port=9901 \
your_program.py <normal cl args> --deepspeed ds_config.json
```

Check out the [Resource Configuration (multi-node)](https://www.deepspeed.ai/getting-started/#resource-configuration-multi-node) guide for more details about configuring multi-node compute resources.

</hfoption>
</hfoptions>

### SLURM

In a SLURM environment, you'll need to adapt your SLURM script to your specific SLURM environment. An example SLURM script may look like:

```bash
#SBATCH --job-name=test-nodes        # name
#SBATCH --nodes=2                    # nodes
#SBATCH --ntasks-per-node=1          # crucial - only 1 task per dist per node!


key: _node_content
string_value:
{"id_": "a64c6cd6-2ccc-4247-b9dc-9f4adf42f59c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5bbd5eaa-4763-4bb6-adc8-96053405c0c6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "f49c03b668f2b0671b2660aa28b7bbb0a4c969e6e7b8a41806b02ed09c25a72e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6b5b2c19-cd56-484e-bed1-0e78c4265904", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "9d3815082be71e613fe8db4e41c119cdc09f4433f904cdc07a65fe71182f18ca", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bb7ba753-0705-41fd-9c87-7fa1896803d7", "node_type": "1", "metadata": {}, "hash": "b6645005c21e1286b0082b2671cdad41dcffc1fd27e1beeb5059d991e6120c12", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 32881, "end_char_idx": 35623, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<hfoptions id="multinode">
<hfoption id="torchrun">

For [torchrun](https://pytorch.org/docs/stable/elastic/run.html), you have to ssh to each node and run the following command on both of them. The launcher waits until both nodes are synchronized before launching the training.

```bash
torchrun --nproc_per_node=8 --nnode=2 --node_rank=0 --master_addr=hostname1 \
--master_port=9901 your_program.py <normal cl args> --deepspeed ds_config.json
```

</hfoption>
<hfoption id="deepspeed">

For the `deepspeed` launcher, start by creating a `hostfile`.

```bash
hostname1 slots=8
hostname2 slots=8
```

Then you can launch the training with the following command. The `deepspeed` launcher automatically launches the command on both nodes at once.

```bash
deepspeed --num_gpus 8 --num_nodes 2 --hostfile hostfile --master_addr hostname1 --master_port=9901 \
your_program.py <normal cl args> --deepspeed ds_config.json
```

Check out the [Resource Configuration (multi-node)](https://www.deepspeed.ai/getting-started/#resource-configuration-multi-node) guide for more details about configuring multi-node compute resources.

</hfoption>
</hfoptions>

### SLURM

In a SLURM environment, you'll need to adapt your SLURM script to your specific SLURM environment. An example SLURM script may look like:

```bash
#SBATCH --job-name=test-nodes        # name
#SBATCH --nodes=2                    # nodes
#SBATCH --ntasks-per-node=1          # crucial - only 1 task per dist per node!
#SBATCH --cpus-per-task=10           # number of cores per tasks
#SBATCH --gres=gpu:8                 # number of gpus
#SBATCH --time 20:00:00              # maximum execution time (HH:MM:SS)
#SBATCH --output=%x-%j.out           # output file name

export GPUS_PER_NODE=8
export MASTER_ADDR=$(scontrol show hostnames $SLURM_JOB_NODELIST | head -n 1)
export MASTER_PORT=9901

srun --jobid $SLURM_JOBID bash -c 'python -m torch.distributed.run \
 --nproc_per_node $GPUS_PER_NODE --nnodes $SLURM_NNODES --node_rank $SLURM_PROCID \
 --master_addr $MASTER_ADDR --master_port $MASTER_PORT \
your_program.py <normal cl args> --deepspeed ds_config.json'
```

Then you can schedule your multi-node deployment with the following command which launches training simultaneously on all nodes.

```bash
sbatch launch.slurm
```

### Notebook

The `deepspeed` launcher doesn't support deployment from a notebook so you'll need to emulate the distributed environment. However, this only works for 1 GPU. If you want to use more than 1 GPU, you must use a multi-process environment for DeepSpeed to work. This means you have to use the `deepspeed` launcher which can't be emulated as shown here.

```py
# DeepSpeed requires a distributed environment even when only one process is used.


key: _node_content
string_value:
{"id_": "bb7ba753-0705-41fd-9c87-7fa1896803d7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5bbd5eaa-4763-4bb6-adc8-96053405c0c6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "f49c03b668f2b0671b2660aa28b7bbb0a4c969e6e7b8a41806b02ed09c25a72e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a64c6cd6-2ccc-4247-b9dc-9f4adf42f59c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "b70595f8055e919ef147e5c0162554095608b332392d6cc8d043b0689a6793a8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7fc8ccb3-0325-4d3e-9ea4-36e3ccbd5d70", "node_type": "1", "metadata": {}, "hash": "ed5fc67b85481cf8284f99a78f496c7902f6281d388dc92ee78a6a4d6187b8db", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 34139, "end_char_idx": 36192, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
An example SLURM script may look like:

```bash
#SBATCH --job-name=test-nodes        # name
#SBATCH --nodes=2                    # nodes
#SBATCH --ntasks-per-node=1          # crucial - only 1 task per dist per node!
#SBATCH --cpus-per-task=10           # number of cores per tasks
#SBATCH --gres=gpu:8                 # number of gpus
#SBATCH --time 20:00:00              # maximum execution time (HH:MM:SS)
#SBATCH --output=%x-%j.out           # output file name

export GPUS_PER_NODE=8
export MASTER_ADDR=$(scontrol show hostnames $SLURM_JOB_NODELIST | head -n 1)
export MASTER_PORT=9901

srun --jobid $SLURM_JOBID bash -c 'python -m torch.distributed.run \
 --nproc_per_node $GPUS_PER_NODE --nnodes $SLURM_NNODES --node_rank $SLURM_PROCID \
 --master_addr $MASTER_ADDR --master_port $MASTER_PORT \
your_program.py <normal cl args> --deepspeed ds_config.json'
```

Then you can schedule your multi-node deployment with the following command which launches training simultaneously on all nodes.

```bash
sbatch launch.slurm
```

### Notebook

The `deepspeed` launcher doesn't support deployment from a notebook so you'll need to emulate the distributed environment. However, this only works for 1 GPU. If you want to use more than 1 GPU, you must use a multi-process environment for DeepSpeed to work. This means you have to use the `deepspeed` launcher which can't be emulated as shown here.

```py
# DeepSpeed requires a distributed environment even when only one process is used.
# This emulates a launcher in the notebook
import os

os.environ["MASTER_ADDR"] = "localhost"
os.environ["MASTER_PORT"] = "9994"  # modify if RuntimeError: Address already in use
os.environ["RANK"] = "0"
os.environ["LOCAL_RANK"] = "0"
os.environ["WORLD_SIZE"] = "1"

# Now proceed as normal, plus pass the DeepSpeed config file
training_args = TrainingArguments(..., deepspeed="ds_config_zero3.json")
trainer = Trainer(...)
trainer.train()
```

If you want to create the config file on the fly in the notebook in the current directory, you could have a dedicated cell.


key: _node_content
string_value:
{"id_": "7fc8ccb3-0325-4d3e-9ea4-36e3ccbd5d70", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5bbd5eaa-4763-4bb6-adc8-96053405c0c6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "f49c03b668f2b0671b2660aa28b7bbb0a4c969e6e7b8a41806b02ed09c25a72e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bb7ba753-0705-41fd-9c87-7fa1896803d7", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "6562b1a1e718bb05d6f9dd6e5a39b9655d757acf54e8b344760b175313d7f5ce", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b6612c4b-cb19-4148-8ece-030a77e366b9", "node_type": "1", "metadata": {}, "hash": "1fc13e4f41510f9748c5999738a99581e93c4d8d01a6fe635d09fee4417bdbae", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 35137, "end_char_idx": 37836, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```bash
sbatch launch.slurm
```

### Notebook

The `deepspeed` launcher doesn't support deployment from a notebook so you'll need to emulate the distributed environment. However, this only works for 1 GPU. If you want to use more than 1 GPU, you must use a multi-process environment for DeepSpeed to work. This means you have to use the `deepspeed` launcher which can't be emulated as shown here.

```py
# DeepSpeed requires a distributed environment even when only one process is used.
# This emulates a launcher in the notebook
import os

os.environ["MASTER_ADDR"] = "localhost"
os.environ["MASTER_PORT"] = "9994"  # modify if RuntimeError: Address already in use
os.environ["RANK"] = "0"
os.environ["LOCAL_RANK"] = "0"
os.environ["WORLD_SIZE"] = "1"

# Now proceed as normal, plus pass the DeepSpeed config file
training_args = TrainingArguments(..., deepspeed="ds_config_zero3.json")
trainer = Trainer(...)
trainer.train()
```

If you want to create the config file on the fly in the notebook in the current directory, you could have a dedicated cell.

```py
%%bash
cat <<'EOT' > ds_config_zero3.json
{
    "fp16": {
        "enabled": "auto",
        "loss_scale": 0,
        "loss_scale_window": 1000,
        "initial_scale_power": 16,
        "hysteresis": 2,
        "min_loss_scale": 1
    },

    "optimizer": {
        "type": "AdamW",
        "params": {
            "lr": "auto",
            "betas": "auto",
            "eps": "auto",
            "weight_decay": "auto"
        }
    },

    "scheduler": {
        "type": "WarmupLR",
        "params": {
            "warmup_min_lr": "auto",
            "warmup_max_lr": "auto",
            "warmup_num_steps": "auto"
        }
    },

    "zero_optimization": {
        "stage": 3,
        "offload_optimizer": {
            "device": "cpu",
            "pin_memory": true
        },
        "offload_param": {
            "device": "cpu",
            "pin_memory": true
        },
        "overlap_comm": true,
        "contiguous_gradients": true,
        "sub_group_size": 1e9,
        "reduce_bucket_size": "auto",
        "stage3_prefetch_bucket_size": "auto",
        "stage3_param_persistence_threshold": "auto",
        "stage3_max_live_parameters": 1e9,
        "stage3_max_reuse_distance": 1e9,
        "stage3_gather_16bit_weights_on_model_save": true
    },

    "gradient_accumulation_steps": "auto",
    "gradient_clipping": "auto",
    "steps_per_print": 2000,
    "train_batch_size": "auto",
    "train_micro_batch_size_per_gpu": "auto",
    "wall_clock_breakdown": false
}
EOT
```

If the training script is in a file and not in a notebook cell, you can launch `deepspeed` normally from the shell in a notebook cell.


key: _node_content
string_value:
{"id_": "b6612c4b-cb19-4148-8ece-030a77e366b9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5bbd5eaa-4763-4bb6-adc8-96053405c0c6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "f49c03b668f2b0671b2660aa28b7bbb0a4c969e6e7b8a41806b02ed09c25a72e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7fc8ccb3-0325-4d3e-9ea4-36e3ccbd5d70", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "edc3d93f0043851ec3149c57aba438d73cc2e520e98474f3071643e66e541690", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e00c43ac-a15f-4e06-a4fc-ef0925c86c37", "node_type": "1", "metadata": {}, "hash": "20888e50e29d7df9606751a329a30ed75fd4e57255c9e46abc727aa5d583ac93", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 37837, "end_char_idx": 40898, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
For example, to launch `run_translation.py`:

```py
!git clone https://github.com/huggingface/transformers
!cd transformers; deepspeed examples/pytorch/translation/run_translation.py ...
```

You could also use `%%bash` magic and write multi-line code to run the shell program, but you won't be able to view the logs until training is complete. With `%%bash` magic, you don't need to emulate a distributed environment.

```py
%%bash

git clone https://github.com/huggingface/transformers
cd transformers
deepspeed examples/pytorch/translation/run_translation.py ...
```

## Save model weights

DeepSpeed stores the main full precision fp32 weights in custom checkpoint optimizer files (the glob pattern looks like `global_step*/*optim_states.pt`) and are saved under the normal checkpoint.

<hfoptions id="save">
<hfoption id="fp16">

A model trained with ZeRO-2 saves the pytorch_model.bin weights in fp16. To save the model weights in fp16 for a model trained with ZeRO-3, you need to set `"stage3_gather_16bit_weights_on_model_save": true` because the model weights are partitioned across multiple GPUs. Otherwise, the [`Trainer`] won't save the weights in fp16 and it won't create a pytorch_model.bin file. This is because DeepSpeed's state_dict contains a placeholder instead of the real weights and you won't be able to load them.

```yaml
{
    "zero_optimization": {
        "stage3_gather_16bit_weights_on_model_save": true
    }
}
```

</hfoption>
<hfoption id="fp32">

The full precision weights shouldn't be saved during training because it can require a lot of memory. It is usually best to save the fp32 weights offline after training is complete. But if you have a lot of free CPU memory, it is possible to save the fp32 weights during training. This section covers both online and offline approaches.

### Online

You must have saved at least one checkpoint to load the latest checkpoint as shown in the following:

```py
from transformers.trainer_utils import get_last_checkpoint
from deepspeed.utils.zero_to_fp32 import load_state_dict_from_zero_checkpoint

checkpoint_dir = get_last_checkpoint(trainer.args.output_dir)
fp32_model = load_state_dict_from_zero_checkpoint(trainer.model, checkpoint_dir)
```

If you've enabled the `--load_best_model_at_end` parameter to track the best checkpoint in [`TrainingArguments`], you can finish training first and save the final model explicitly. Then you can reload it as shown below:

```py
from deepspeed.utils.zero_to_fp32 import load_state_dict_from_zero_checkpoint

checkpoint_dir = os.path.join(trainer.args.output_dir, "checkpoint-final")
trainer.deepspeed.save_checkpoint(checkpoint_dir)
fp32_model = load_state_dict_from_zero_checkpoint(trainer.model, checkpoint_dir)
```

<Tip>

Once `load_state_dict_from_zero_checkpoint` is run, the model is no longer usable in DeepSpeed in the context of the same application. You'll need to initialize the DeepSpeed engine again since `model.load_state_dict(state_dict)` removes all the DeepSpeed magic from it. Only use this at the very end of training.


key: _node_content
string_value:
{"id_": "e00c43ac-a15f-4e06-a4fc-ef0925c86c37", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5bbd5eaa-4763-4bb6-adc8-96053405c0c6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "f49c03b668f2b0671b2660aa28b7bbb0a4c969e6e7b8a41806b02ed09c25a72e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b6612c4b-cb19-4148-8ece-030a77e366b9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "91e1481a5728aaf3be9f1a57d6fb924500cc8726bc93cd421a3a52df7e34c817", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ea84ebfa-220a-4a95-95a7-f1c6e6d4669a", "node_type": "1", "metadata": {}, "hash": "51c3cdc4c60328e39e9f3d56d940885bb0aba06d345681fb5bd03b6515cc8961", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 39175, "end_char_idx": 41430, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```yaml
{
    "zero_optimization": {
        "stage3_gather_16bit_weights_on_model_save": true
    }
}
```

</hfoption>
<hfoption id="fp32">

The full precision weights shouldn't be saved during training because it can require a lot of memory. It is usually best to save the fp32 weights offline after training is complete. But if you have a lot of free CPU memory, it is possible to save the fp32 weights during training. This section covers both online and offline approaches.

### Online

You must have saved at least one checkpoint to load the latest checkpoint as shown in the following:

```py
from transformers.trainer_utils import get_last_checkpoint
from deepspeed.utils.zero_to_fp32 import load_state_dict_from_zero_checkpoint

checkpoint_dir = get_last_checkpoint(trainer.args.output_dir)
fp32_model = load_state_dict_from_zero_checkpoint(trainer.model, checkpoint_dir)
```

If you've enabled the `--load_best_model_at_end` parameter to track the best checkpoint in [`TrainingArguments`], you can finish training first and save the final model explicitly. Then you can reload it as shown below:

```py
from deepspeed.utils.zero_to_fp32 import load_state_dict_from_zero_checkpoint

checkpoint_dir = os.path.join(trainer.args.output_dir, "checkpoint-final")
trainer.deepspeed.save_checkpoint(checkpoint_dir)
fp32_model = load_state_dict_from_zero_checkpoint(trainer.model, checkpoint_dir)
```

<Tip>

Once `load_state_dict_from_zero_checkpoint` is run, the model is no longer usable in DeepSpeed in the context of the same application. You'll need to initialize the DeepSpeed engine again since `model.load_state_dict(state_dict)` removes all the DeepSpeed magic from it. Only use this at the very end of training.

</Tip>

You can also extract and load the state_dict of the fp32 weights:

```py
from deepspeed.utils.zero_to_fp32 import get_fp32_state_dict_from_zero_checkpoint

state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir)  # already on cpu
model = model.cpu()
model.load_state_dict(state_dict)
```

### Offline

DeepSpeed provides a zero_to_fp32.py script at the top-level of the checkpoint folder for extracting weights at any point. This is a standalone script and you don't need a configuration file or [`Trainer`].


key: _node_content
string_value:
{"id_": "ea84ebfa-220a-4a95-95a7-f1c6e6d4669a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5bbd5eaa-4763-4bb6-adc8-96053405c0c6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "f49c03b668f2b0671b2660aa28b7bbb0a4c969e6e7b8a41806b02ed09c25a72e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e00c43ac-a15f-4e06-a4fc-ef0925c86c37", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "bc6250ad915409cdda736c156bdfa9c06fd86a8304fe91aec8998406671e13c1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ca05393b-c275-4a02-8f1b-98e4bcdf539a", "node_type": "1", "metadata": {}, "hash": "7f929c4725a79004d526026ec489672acf83bdc611dc090fbe4379bb3b0d1971", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 39598, "end_char_idx": 42455, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This section covers both online and offline approaches.

### Online

You must have saved at least one checkpoint to load the latest checkpoint as shown in the following:

```py
from transformers.trainer_utils import get_last_checkpoint
from deepspeed.utils.zero_to_fp32 import load_state_dict_from_zero_checkpoint

checkpoint_dir = get_last_checkpoint(trainer.args.output_dir)
fp32_model = load_state_dict_from_zero_checkpoint(trainer.model, checkpoint_dir)
```

If you've enabled the `--load_best_model_at_end` parameter to track the best checkpoint in [`TrainingArguments`], you can finish training first and save the final model explicitly. Then you can reload it as shown below:

```py
from deepspeed.utils.zero_to_fp32 import load_state_dict_from_zero_checkpoint

checkpoint_dir = os.path.join(trainer.args.output_dir, "checkpoint-final")
trainer.deepspeed.save_checkpoint(checkpoint_dir)
fp32_model = load_state_dict_from_zero_checkpoint(trainer.model, checkpoint_dir)
```

<Tip>

Once `load_state_dict_from_zero_checkpoint` is run, the model is no longer usable in DeepSpeed in the context of the same application. You'll need to initialize the DeepSpeed engine again since `model.load_state_dict(state_dict)` removes all the DeepSpeed magic from it. Only use this at the very end of training.

</Tip>

You can also extract and load the state_dict of the fp32 weights:

```py
from deepspeed.utils.zero_to_fp32 import get_fp32_state_dict_from_zero_checkpoint

state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir)  # already on cpu
model = model.cpu()
model.load_state_dict(state_dict)
```

### Offline

DeepSpeed provides a zero_to_fp32.py script at the top-level of the checkpoint folder for extracting weights at any point. This is a standalone script and you don't need a configuration file or [`Trainer`].

For example, if your checkpoint folder looked like this:

```bash
$ ls -l output_dir/checkpoint-1/
-rw-rw-r-- 1 stas stas 1.4K Mar 27 20:42 config.json
drwxrwxr-x 2 stas stas 4.0K Mar 25 19:52 global_step1/
-rw-rw-r-- 1 stas stas   12 Mar 27 13:16 latest
-rw-rw-r-- 1 stas stas 827K Mar 27 20:42 optimizer.pt
-rw-rw-r-- 1 stas stas 231M Mar 27 20:42 pytorch_model.bin
-rw-rw-r-- 1 stas stas  623 Mar 27 20:42 scheduler.pt
-rw-rw-r-- 1 stas stas 1.8K Mar 27 20:42 special_tokens_map.json
-rw-rw-r-- 1 stas stas 774K Mar 27 20:42 spiece.model
-rw-rw-r-- 1 stas stas 1.9K Mar 27 20:42 tokenizer_config.json
-rw-rw-r-- 1 stas stas  339 Mar 27 20:42 trainer_state.json
-rw-rw-r-- 1 stas stas 2.3K Mar 27 20:42 training_args.bin
-rwxrw-r-- 1 stas stas 5.5K Mar 27 13:16 zero_to_fp32.py*
```

To reconstruct the fp32 weights from the DeepSpeed checkpoint (ZeRO-2 or ZeRO-3) subfolder `global_step1`, run the following command to create and consolidate the full fp32 weights from multiple GPUs into a single pytorch_model.bin file.


key: _node_content
string_value:
{"id_": "ca05393b-c275-4a02-8f1b-98e4bcdf539a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5bbd5eaa-4763-4bb6-adc8-96053405c0c6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "f49c03b668f2b0671b2660aa28b7bbb0a4c969e6e7b8a41806b02ed09c25a72e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ea84ebfa-220a-4a95-95a7-f1c6e6d4669a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "86b7c0c13dfbffca53b68a5944efb486535b1e23dc64c92d555add97d07d7d8c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a9b22bb3-4616-409e-ae1f-72a4f7120b9e", "node_type": "1", "metadata": {}, "hash": "3cf560e95490b20f80fddaaba9ac8641a7b08ab155af421cbdacb264135fc3eb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 42456, "end_char_idx": 45633, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The script automatically discovers the subfolder containing the checkpoint.

```py
python zero_to_fp32.py . pytorch_model.bin
```

<Tip>

Run `python zero_to_fp32.py -h` for more usage details. The script requires 2x the general RAM of the final fp32 weights.

</Tip>

</hfoption>
</hfoptions>

## ZeRO Inference

[ZeRO Inference](https://www.deepspeed.ai/2022/09/09/zero-inference.html) places the model weights in CPU or NVMe memory to avoid burdening the GPU which makes it possible to run inference with huge models on a GPU. Inference doesn't require any large additional amounts of memory for the optimizer states and gradients so you can fit much larger batches and/or sequence lengths on the same hardware.

ZeRO Inference shares the same configuration file as [ZeRO-3](#zero-configuration), and ZeRO-2 and ZeRO-1 configs won't work because they don't provide any benefits for inference.

To run ZeRO Inference, pass your usual training arguments to the [`TrainingArguments`] class and add the `--do_eval` argument.

```bash
deepspeed --num_gpus=2 your_program.py <normal cl args> --do_eval --deepspeed ds_config.json
```

## Non-Trainer DeepSpeed integration

DeepSpeed also works with Transformers without the [`Trainer`] class. This is handled by the [`HfDeepSpeedConfig`] which only takes care of gathering ZeRO-3 parameters and splitting a model across multiple GPUs when you call [`~PreTrainedModel.from_pretrained`].

<Tip>

If you want everything automatically taken care of for you, try using DeepSpeed with the [`Trainer`]! You'll need to follow the [DeepSpeed documentation](https://www.deepspeed.ai/), and manually configure the parameter values in the config file (you can't use the `"auto"` value).

</Tip>

To efficiently deploy ZeRO-3, you must instantiate the [`HfDeepSpeedConfig`] object before the model and keep that object alive:

<hfoptions id="models">
<hfoption id="pretrained model">

```py
from transformers.integrations import HfDeepSpeedConfig
from transformers import AutoModel
import deepspeed

ds_config = {...}  # deepspeed config object or path to the file
# must run before instantiating the model to detect zero 3
dschf = HfDeepSpeedConfig(ds_config)  # keep this object alive
model = AutoModel.from_pretrained("openai-community/gpt2")
engine = deepspeed.initialize(model=model, config_params=ds_config, ...)
```

</hfoption>
<hfoption id="non-pretrained model">

[`HfDeepSpeedConfig`] is not required for ZeRO-1 or ZeRO-2.

```py
from transformers.integrations import HfDeepSpeedConfig
from transformers import AutoModel, AutoConfig
import deepspeed

ds_config = {...}  # deepspeed config object or path to the file
# must run before instantiating the model to detect zero 3
dschf = HfDeepSpeedConfig(ds_config)  # keep this object alive
config = AutoConfig.from_pretrained("openai-community/gpt2")
model = AutoModel.from_config(config)
engine = deepspeed.initialize(model=model, config_params=ds_config, ...)
```

</hfoption>
</hfoptions>

### Non-Trainer ZeRO Inference

To run ZeRO Inference without the [`Trainer`] in cases where you can’t fit a model onto a single GPU, try using additional GPUs or/and offloading to CPU memory.


key: _node_content
string_value:
{"id_": "a9b22bb3-4616-409e-ae1f-72a4f7120b9e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5bbd5eaa-4763-4bb6-adc8-96053405c0c6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "f49c03b668f2b0671b2660aa28b7bbb0a4c969e6e7b8a41806b02ed09c25a72e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ca05393b-c275-4a02-8f1b-98e4bcdf539a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "7c0d9fc2d8ca89556642ec4782fc20c488c9010606b14594b6071e5b783cbe60", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a76c46ae-efb5-4248-93c1-a4d4de0ea5c0", "node_type": "1", "metadata": {}, "hash": "bc502238b44027b2e5282dd3a487ae1c8d8123140144a8359ae32f17957e2782", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 44178, "end_char_idx": 47184, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>

To efficiently deploy ZeRO-3, you must instantiate the [`HfDeepSpeedConfig`] object before the model and keep that object alive:

<hfoptions id="models">
<hfoption id="pretrained model">

```py
from transformers.integrations import HfDeepSpeedConfig
from transformers import AutoModel
import deepspeed

ds_config = {...}  # deepspeed config object or path to the file
# must run before instantiating the model to detect zero 3
dschf = HfDeepSpeedConfig(ds_config)  # keep this object alive
model = AutoModel.from_pretrained("openai-community/gpt2")
engine = deepspeed.initialize(model=model, config_params=ds_config, ...)
```

</hfoption>
<hfoption id="non-pretrained model">

[`HfDeepSpeedConfig`] is not required for ZeRO-1 or ZeRO-2.

```py
from transformers.integrations import HfDeepSpeedConfig
from transformers import AutoModel, AutoConfig
import deepspeed

ds_config = {...}  # deepspeed config object or path to the file
# must run before instantiating the model to detect zero 3
dschf = HfDeepSpeedConfig(ds_config)  # keep this object alive
config = AutoConfig.from_pretrained("openai-community/gpt2")
model = AutoModel.from_config(config)
engine = deepspeed.initialize(model=model, config_params=ds_config, ...)
```

</hfoption>
</hfoptions>

### Non-Trainer ZeRO Inference

To run ZeRO Inference without the [`Trainer`] in cases where you can’t fit a model onto a single GPU, try using additional GPUs or/and offloading to CPU memory. The important nuance to understand here is that the way ZeRO is designed, you can process different inputs on different GPUs in parallel.

Make sure to:

* disable CPU offload if you have enough GPU memory (since it slows things down).
* enable bf16 if you have an Ampere or newer GPU to make things faster. If you don’t have one of these GPUs, you may enable fp16 as long as you don’t use a model pretrained in bf16 (T5 models) because it may lead to an overflow error.

Take a look at the following script to get a better idea of how to run ZeRO Inference without the [`Trainer`] on a model that won't fit on a single GPU.

```py
#!/usr/bin/env python

# This script demonstrates how to use Deepspeed ZeRO in an inference mode when one can't fit a model
# into a single GPU
#
# 1. Use 1 GPU with CPU offload
# 2. Or use multiple GPUs instead
#
# First you need to install deepspeed: pip install deepspeed
#
# Here we use a 3B "bigscience/T0_3B" model which needs about 15GB GPU RAM - so 1 largish or 2
# small GPUs can handle it. or 1 small GPU and a lot of CPU memory.
#
# To use a larger model like "bigscience/T0" which needs about 50GB, unless you have an 80GB GPU -
# you will need 2-4 gpus. And then you can adapt the script to handle more gpus if you want to
# process multiple inputs at once.
#
# The provided deepspeed config also activates CPU memory offloading, so chances are that if you
# have a lot of available CPU memory and you don't mind a slowdown you should be able to load a
# model that doesn't normally fit into a single GPU.


key: _node_content
string_value:
{"id_": "a76c46ae-efb5-4248-93c1-a4d4de0ea5c0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5bbd5eaa-4763-4bb6-adc8-96053405c0c6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "f49c03b668f2b0671b2660aa28b7bbb0a4c969e6e7b8a41806b02ed09c25a72e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a9b22bb3-4616-409e-ae1f-72a4f7120b9e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "3ff8b40bb4341160681ffe2b8cb02ead7457ca387da1ea443b0d9a9e9a11bc74", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "69c90725-5da3-4310-a52c-959d19f7e9eb", "node_type": "1", "metadata": {}, "hash": "7415ba0d0d9abd3b32a52e28afbde2a793d58b977230237603673bf09f5e7749", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 45773, "end_char_idx": 48464, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Make sure to:

* disable CPU offload if you have enough GPU memory (since it slows things down).
* enable bf16 if you have an Ampere or newer GPU to make things faster. If you don’t have one of these GPUs, you may enable fp16 as long as you don’t use a model pretrained in bf16 (T5 models) because it may lead to an overflow error.

Take a look at the following script to get a better idea of how to run ZeRO Inference without the [`Trainer`] on a model that won't fit on a single GPU.

```py
#!/usr/bin/env python

# This script demonstrates how to use Deepspeed ZeRO in an inference mode when one can't fit a model
# into a single GPU
#
# 1. Use 1 GPU with CPU offload
# 2. Or use multiple GPUs instead
#
# First you need to install deepspeed: pip install deepspeed
#
# Here we use a 3B "bigscience/T0_3B" model which needs about 15GB GPU RAM - so 1 largish or 2
# small GPUs can handle it. or 1 small GPU and a lot of CPU memory.
#
# To use a larger model like "bigscience/T0" which needs about 50GB, unless you have an 80GB GPU -
# you will need 2-4 gpus. And then you can adapt the script to handle more gpus if you want to
# process multiple inputs at once.
#
# The provided deepspeed config also activates CPU memory offloading, so chances are that if you
# have a lot of available CPU memory and you don't mind a slowdown you should be able to load a
# model that doesn't normally fit into a single GPU. If you have enough GPU memory the program will
# run faster if you don't want offload to CPU - so disable that section then.
#
# To deploy on 1 gpu:
#
# deepspeed --num_gpus 1 t0.py
# or:
# python -m torch.distributed.run --nproc_per_node=1 t0.py
#
# To deploy on 2 gpus:
#
# deepspeed --num_gpus 2 t0.py
# or:
# python -m torch.distributed.run --nproc_per_node=2 t0.py

from transformers import AutoTokenizer, AutoConfig, AutoModelForSeq2SeqLM
from transformers.integrations import HfDeepSpeedConfig
import deepspeed
import os
import torch

os.environ["TOKENIZERS_PARALLELISM"] = "false"  # To avoid warnings about parallelism in tokenizers

# distributed setup
local_rank = int(os.getenv("LOCAL_RANK", "0"))
world_size = int(os.getenv("WORLD_SIZE", "1"))
torch.cuda.set_device(local_rank)
deepspeed.init_distributed()

model_name = "bigscience/T0_3B"

config = AutoConfig.from_pretrained(model_name)
model_hidden_size = config.d_model

# batch size has to be divisible by world_size, but can be bigger than world_size
train_batch_size = 1 * world_size

# ds_config notes
#
# - enable bf16 if you use Ampere or higher GPU - this will run in mixed precision and will be
# faster.
#
# - for older GPUs you can enable fp16, but it'll only work for non-bf16 pretrained models - e.g.


key: _node_content
string_value:
{"id_": "69c90725-5da3-4310-a52c-959d19f7e9eb", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5bbd5eaa-4763-4bb6-adc8-96053405c0c6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "f49c03b668f2b0671b2660aa28b7bbb0a4c969e6e7b8a41806b02ed09c25a72e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a76c46ae-efb5-4248-93c1-a4d4de0ea5c0", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "ce676a46278da14be1c0bba3aff81efec0b8b227bfd9da4961679f46ea6ccda5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e03d0f92-6011-4b7a-8041-445921534c51", "node_type": "1", "metadata": {}, "hash": "d3bf100c4b6cdc92113609c0209b02f97789cf74ea8d4f6e95dd1ad6fcc492fd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 47185, "end_char_idx": 49918, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
If you have enough GPU memory the program will
# run faster if you don't want offload to CPU - so disable that section then.
#
# To deploy on 1 gpu:
#
# deepspeed --num_gpus 1 t0.py
# or:
# python -m torch.distributed.run --nproc_per_node=1 t0.py
#
# To deploy on 2 gpus:
#
# deepspeed --num_gpus 2 t0.py
# or:
# python -m torch.distributed.run --nproc_per_node=2 t0.py

from transformers import AutoTokenizer, AutoConfig, AutoModelForSeq2SeqLM
from transformers.integrations import HfDeepSpeedConfig
import deepspeed
import os
import torch

os.environ["TOKENIZERS_PARALLELISM"] = "false"  # To avoid warnings about parallelism in tokenizers

# distributed setup
local_rank = int(os.getenv("LOCAL_RANK", "0"))
world_size = int(os.getenv("WORLD_SIZE", "1"))
torch.cuda.set_device(local_rank)
deepspeed.init_distributed()

model_name = "bigscience/T0_3B"

config = AutoConfig.from_pretrained(model_name)
model_hidden_size = config.d_model

# batch size has to be divisible by world_size, but can be bigger than world_size
train_batch_size = 1 * world_size

# ds_config notes
#
# - enable bf16 if you use Ampere or higher GPU - this will run in mixed precision and will be
# faster.
#
# - for older GPUs you can enable fp16, but it'll only work for non-bf16 pretrained models - e.g.
# all official t5 models are bf16-pretrained
#
# - set offload_param.device to "none" or completely remove the `offload_param` section if you don't
# - want CPU offload
#
# - if using `offload_param` you can manually finetune stage3_param_persistence_threshold to control
# - which params should remain on gpus - the larger the value the smaller the offload size
#
# For in-depth info on Deepspeed config see
# https://huggingface.co/docs/transformers/main/main_classes/deepspeed

# keeping the same format as json for consistency, except it uses lower case for true/false
# fmt: off
ds_config = {
    "fp16": {
        "enabled": False
    },
    "bf16": {
        "enabled": False
    },
    "zero_optimization": {
        "stage": 3,
        "offload_param": {
            "device": "cpu",
            "pin_memory": True
        },
        "overlap_comm": True,
        "contiguous_gradients": True,
        "reduce_bucket_size": model_hidden_size * model_hidden_size,
        "stage3_prefetch_bucket_size": 0.9 * model_hidden_size * model_hidden_size,
        "stage3_param_persistence_threshold": 10 * model_hidden_size
    },
    "steps_per_print": 2000,
    "train_batch_size": train_batch_size,
    "train_micro_batch_size_per_gpu": 1,
    "wall_clock_breakdown": False
}
# fmt: on

# next line instructs transformers to partition the model directly over multiple gpus using
# deepspeed.zero.Init when model's `from_pretrained` method is called.


key: _node_content
string_value:
{"id_": "e03d0f92-6011-4b7a-8041-445921534c51", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5bbd5eaa-4763-4bb6-adc8-96053405c0c6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "f49c03b668f2b0671b2660aa28b7bbb0a4c969e6e7b8a41806b02ed09c25a72e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "69c90725-5da3-4310-a52c-959d19f7e9eb", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "e2d9f0cae8cd71625fa7ca153ed4e04a5df70fb35857436c2f846ce7453e0c6b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ddbb91b3-96c0-44f1-bbd5-5c8d0a10b405", "node_type": "1", "metadata": {}, "hash": "f992ddf2a684638556da25a46c2ccf32caa85cf42436590352c56838ab0cb6e0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 48365, "end_char_idx": 51041, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
#
# - for older GPUs you can enable fp16, but it'll only work for non-bf16 pretrained models - e.g.
# all official t5 models are bf16-pretrained
#
# - set offload_param.device to "none" or completely remove the `offload_param` section if you don't
# - want CPU offload
#
# - if using `offload_param` you can manually finetune stage3_param_persistence_threshold to control
# - which params should remain on gpus - the larger the value the smaller the offload size
#
# For in-depth info on Deepspeed config see
# https://huggingface.co/docs/transformers/main/main_classes/deepspeed

# keeping the same format as json for consistency, except it uses lower case for true/false
# fmt: off
ds_config = {
    "fp16": {
        "enabled": False
    },
    "bf16": {
        "enabled": False
    },
    "zero_optimization": {
        "stage": 3,
        "offload_param": {
            "device": "cpu",
            "pin_memory": True
        },
        "overlap_comm": True,
        "contiguous_gradients": True,
        "reduce_bucket_size": model_hidden_size * model_hidden_size,
        "stage3_prefetch_bucket_size": 0.9 * model_hidden_size * model_hidden_size,
        "stage3_param_persistence_threshold": 10 * model_hidden_size
    },
    "steps_per_print": 2000,
    "train_batch_size": train_batch_size,
    "train_micro_batch_size_per_gpu": 1,
    "wall_clock_breakdown": False
}
# fmt: on

# next line instructs transformers to partition the model directly over multiple gpus using
# deepspeed.zero.Init when model's `from_pretrained` method is called.
#
# **it has to be run before loading the model AutoModelForSeq2SeqLM.from_pretrained(model_name)**
#
# otherwise the model will first be loaded normally and only partitioned at forward time which is
# less efficient and when there is little CPU RAM may fail
dschf = HfDeepSpeedConfig(ds_config)  # keep this object alive

# now a model can be loaded.
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)

# initialise Deepspeed ZeRO and store only the engine object
ds_engine = deepspeed.initialize(model=model, config_params=ds_config)[0]
ds_engine.module.eval()  # inference

# Deepspeed ZeRO can process unrelated inputs on each GPU. So for 2 gpus you process 2 inputs at once.
# If you use more GPUs adjust for more.
# And of course if you have just one input to process you then need to pass the same string to both gpus
# If you use only one GPU, then you will have only rank 0.
rank = torch.distributed.get_rank()
if rank == 0:
    text_in = "Is this review positive or negative? Review: this is the best cast iron skillet you will ever buy"
elif rank == 1:
    text_in = "Is this review positive or negative?


key: _node_content
string_value:
{"id_": "ddbb91b3-96c0-44f1-bbd5-5c8d0a10b405", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5bbd5eaa-4763-4bb6-adc8-96053405c0c6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "f49c03b668f2b0671b2660aa28b7bbb0a4c969e6e7b8a41806b02ed09c25a72e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e03d0f92-6011-4b7a-8041-445921534c51", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "9b662c6fd94253a44e8a4070654f683818415240919573e0a1127de9bdf4d113", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "191e9e6b-4956-414a-b0c7-4149e1e5580a", "node_type": "1", "metadata": {}, "hash": "6d6b78dfbb1294847f21e9191f445c51d71268caba472c165cafab67fc07bdc8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 49919, "end_char_idx": 52815, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
#
# **it has to be run before loading the model AutoModelForSeq2SeqLM.from_pretrained(model_name)**
#
# otherwise the model will first be loaded normally and only partitioned at forward time which is
# less efficient and when there is little CPU RAM may fail
dschf = HfDeepSpeedConfig(ds_config)  # keep this object alive

# now a model can be loaded.
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)

# initialise Deepspeed ZeRO and store only the engine object
ds_engine = deepspeed.initialize(model=model, config_params=ds_config)[0]
ds_engine.module.eval()  # inference

# Deepspeed ZeRO can process unrelated inputs on each GPU. So for 2 gpus you process 2 inputs at once.
# If you use more GPUs adjust for more.
# And of course if you have just one input to process you then need to pass the same string to both gpus
# If you use only one GPU, then you will have only rank 0.
rank = torch.distributed.get_rank()
if rank == 0:
    text_in = "Is this review positive or negative? Review: this is the best cast iron skillet you will ever buy"
elif rank == 1:
    text_in = "Is this review positive or negative? Review: this is the worst restaurant ever"

tokenizer = AutoTokenizer.from_pretrained(model_name)
inputs = tokenizer.encode(text_in, return_tensors="pt").to(device=local_rank)
with torch.no_grad():
    outputs = ds_engine.module.generate(inputs, synced_gpus=True)
text_out = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(f"rank{rank}:\n   in={text_in}\n  out={text_out}")
```

Save the script as t0.py and launch it:

```bash
$ deepspeed --num_gpus 2 t0.py
rank0:
   in=Is this review positive or negative? Review: this is the best cast iron skillet you will ever buy
  out=Positive
rank1:
   in=Is this review positive or negative? Review: this is the worst restaurant ever
  out=negative
```

This is a very basic example and you'll want to adapt it to your use case.

### Generate

Using multiple GPUs with ZeRO-3 for generation requires synchronizing the GPUs by setting `synced_gpus=True` in the [`~GenerationMixin.generate`] method. Otherwise, if one GPU is finished generating before another one, the whole system hangs because the remaining GPUs haven't received the weight shard from the GPU that finished first.

For Transformers>=4.28, if `synced_gpus` is automatically set to `True` if multiple GPUs are detected during generation.

## Troubleshoot

When you encounter an issue, you should consider whether DeepSpeed is the cause of the problem because often it isn't (unless it's super obviously and you can see DeepSpeed modules in the exception)! The first step should be to retry your setup without DeepSpeed, and if the problem persists, then you can report the issue. If the issue is a core DeepSpeed problem and unrelated to the Transformers integration, open an Issue on the [DeepSpeed repository](https://github.com/microsoft/DeepSpeed).


key: _node_content
string_value:
{"id_": "191e9e6b-4956-414a-b0c7-4149e1e5580a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5bbd5eaa-4763-4bb6-adc8-96053405c0c6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "f49c03b668f2b0671b2660aa28b7bbb0a4c969e6e7b8a41806b02ed09c25a72e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ddbb91b3-96c0-44f1-bbd5-5c8d0a10b405", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "c2142292c0a462fea1dad653954eb84d14651f0d0da7ac8d31388b31da70f423", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "03819fa5-4978-4ea2-b269-5a170c556be2", "node_type": "1", "metadata": {}, "hash": "ed1c51766d2bd15f242f2e480c2ac869014719c80c6109070d5d6f4646b34ebe", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 51564, "end_char_idx": 54620, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Review: this is the best cast iron skillet you will ever buy
  out=Positive
rank1:
   in=Is this review positive or negative? Review: this is the worst restaurant ever
  out=negative
```

This is a very basic example and you'll want to adapt it to your use case.

### Generate

Using multiple GPUs with ZeRO-3 for generation requires synchronizing the GPUs by setting `synced_gpus=True` in the [`~GenerationMixin.generate`] method. Otherwise, if one GPU is finished generating before another one, the whole system hangs because the remaining GPUs haven't received the weight shard from the GPU that finished first.

For Transformers>=4.28, if `synced_gpus` is automatically set to `True` if multiple GPUs are detected during generation.

## Troubleshoot

When you encounter an issue, you should consider whether DeepSpeed is the cause of the problem because often it isn't (unless it's super obviously and you can see DeepSpeed modules in the exception)! The first step should be to retry your setup without DeepSpeed, and if the problem persists, then you can report the issue. If the issue is a core DeepSpeed problem and unrelated to the Transformers integration, open an Issue on the [DeepSpeed repository](https://github.com/microsoft/DeepSpeed).

For issues related to the Transformers integration, please provide the following information:

* the full DeepSpeed config file

* the command line arguments of the [`Trainer`], or [`TrainingArguments`] arguments if you're scripting the [`Trainer`] setup yourself (don't dump the [`TrainingArguments`] which has dozens of irrelevant entries)

* the outputs of:

```bash
python -c 'import torch; print(f"torch: {torch.__version__}")'
python -c 'import transformers; print(f"transformers: {transformers.__version__}")'
python -c 'import deepspeed; print(f"deepspeed: {deepspeed.__version__}")'
```

* a link to a Google Colab notebook to reproduce the issue

* if impossible, a standard and non-custom dataset we can use and also try to use an existing example to reproduce the issue with

The following sections provide a guide for resolving two of the most common issues.

### DeepSpeed process killed at startup

When the DeepSpeed process is killed during launch without a traceback, that usually means the program tried to allocate more CPU memory than your system has or your process tried to allocate more CPU memory than allowed leading the OS kernel to terminate the process. In this case, check whether your configuration file has either `offload_optimizer`, `offload_param` or both configured to offload to the CPU. 

If you have NVMe and ZeRO-3 setup, experiment with offloading to the NVMe ([estimate](https://deepspeed.readthedocs.io/en/latest/memory.html) the memory requirements for your model).

### NaN loss

NaN loss often occurs when a model is pretrained in bf16 and then you try to use it with fp16 (especially relevant for TPU trained models). To resolve this, use fp32 or bf16 if your hardware supports it (TPU, Ampere GPUs or newer).

The other issue may be related to using fp16.


key: _node_content
string_value:
{"id_": "03819fa5-4978-4ea2-b269-5a170c556be2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5bbd5eaa-4763-4bb6-adc8-96053405c0c6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "f49c03b668f2b0671b2660aa28b7bbb0a4c969e6e7b8a41806b02ed09c25a72e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "191e9e6b-4956-414a-b0c7-4149e1e5580a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "badceb67d5bc29c3c06a7eb65487e63c741edf9482c872f6c03ff18cbdcec81e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "dab006ac-1570-4995-8405-c4947dfd3f3f", "node_type": "1", "metadata": {}, "hash": "155d9b4dc92b350674f861a8bc9dbf32eb14ce6e8be60d042b6ef57112deaee2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 53690, "end_char_idx": 56633, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### DeepSpeed process killed at startup

When the DeepSpeed process is killed during launch without a traceback, that usually means the program tried to allocate more CPU memory than your system has or your process tried to allocate more CPU memory than allowed leading the OS kernel to terminate the process. In this case, check whether your configuration file has either `offload_optimizer`, `offload_param` or both configured to offload to the CPU. 

If you have NVMe and ZeRO-3 setup, experiment with offloading to the NVMe ([estimate](https://deepspeed.readthedocs.io/en/latest/memory.html) the memory requirements for your model).

### NaN loss

NaN loss often occurs when a model is pretrained in bf16 and then you try to use it with fp16 (especially relevant for TPU trained models). To resolve this, use fp32 or bf16 if your hardware supports it (TPU, Ampere GPUs or newer).

The other issue may be related to using fp16. For example, if this is your fp16 configuration:

```yaml
{
    "fp16": {
        "enabled": "auto",
        "loss_scale": 0,
        "loss_scale_window": 1000,
        "initial_scale_power": 16,
        "hysteresis": 2,
        "min_loss_scale": 1
    }
}
```

You might see the following `OVERFLOW!` messages in the logs:

```bash
0%|                                                                                                                             | 0/189 [00:00<?, ?it/s]
 [deepscale] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 262144, reducing to 262144
  1%|▌                                                                                                                    | 1/189 [00:00<01:26,  2.17it/s]
 [deepscale] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 262144, reducing to 131072.0
  1%|█▏
 [...]
 [deepscale] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 1, reducing to 1
 14%|████████████████▌                                                                                                   | 27/189 [00:14<01:13,  2.21it/s]
 [deepscale] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 1, reducing to 1
 15%|█████████████████▏                                                                                                  | 28/189 [00:14<01:13,  2.18it/s]
 [deepscale] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 1, reducing to 1
 15%|█████████████████▊                                                                                                  | 29/189 [00:15<01:13,  2.18it/s]
 [deepscale] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 1, reducing to 1
[...]
```

This means the DeepSpeed loss scaler is unable to find a scaling coefficient to overcome loss overflow. To fix it, try a higher `initial_scale_power` value (32 usually works).

## Resources

DeepSpeed ZeRO is a powerful technology for training and loading very large models for inference with limited GPU resources, making it more accessible to everyone.


key: _node_content
string_value:
{"id_": "dab006ac-1570-4995-8405-c4947dfd3f3f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5bbd5eaa-4763-4bb6-adc8-96053405c0c6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "f49c03b668f2b0671b2660aa28b7bbb0a4c969e6e7b8a41806b02ed09c25a72e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "03819fa5-4978-4ea2-b269-5a170c556be2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/deepspeed", "title": "DeepSpeed", "tokens": 14117, "retrieve_doc": false, "source": "transformers"}, "hash": "596d3449f6afebf89c523a9b84043797e4a33bfa8af64cbb188220f7ff252b4c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 55130, "end_char_idx": 57280, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Rank 0 Skipping step. Attempted loss scale: 262144, reducing to 262144
  1%|▌                                                                                                                    | 1/189 [00:00<01:26,  2.17it/s]
 [deepscale] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 262144, reducing to 131072.0
  1%|█▏
 [...]
 [deepscale] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 1, reducing to 1
 14%|████████████████▌                                                                                                   | 27/189 [00:14<01:13,  2.21it/s]
 [deepscale] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 1, reducing to 1
 15%|█████████████████▏                                                                                                  | 28/189 [00:14<01:13,  2.18it/s]
 [deepscale] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 1, reducing to 1
 15%|█████████████████▊                                                                                                  | 29/189 [00:15<01:13,  2.18it/s]
 [deepscale] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 1, reducing to 1
[...]
```

This means the DeepSpeed loss scaler is unable to find a scaling coefficient to overcome loss overflow. To fix it, try a higher `initial_scale_power` value (32 usually works).

## Resources

DeepSpeed ZeRO is a powerful technology for training and loading very large models for inference with limited GPU resources, making it more accessible to everyone. To learn more about DeepSpeed, feel free to read the [blog posts](https://www.microsoft.com/en-us/research/search/?q=deepspeed), [documentation](https://www.deepspeed.ai/getting-started/), and [GitHub repository](https://github.com/microsoft/deepspeed). 

The following papers are also a great resource for learning more about ZeRO:

* [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://hf.co/papers/1910.02054)
* [ZeRO-Offload: Democratizing Billion-Scale Model Training](https://hf.co/papers/2101.06840)
* [ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning](https://hf.co/papers/2104.07857)


key: _node_content
string_value:
{"id_": "30e4aa16-77cb-4efb-926f-a05ec5bcb73e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "413c3208-e79b-49bf-9253-e391fd1280b2", "node_type": "1", "metadata": {}, "hash": "4bfdfc4013aa0effcda4de666acc0791525346a31cc0c328fa5f3c808473584d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3402, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# 🤗 Transformers

State-of-the-art Machine Learning for [PyTorch](https://pytorch.org/), [TensorFlow](https://www.tensorflow.org/), and [JAX](https://jax.readthedocs.io/en/latest/).

🤗 Transformers provides APIs and tools to easily download and train state-of-the-art pretrained models. Using pretrained models can reduce your compute costs, carbon footprint, and save you the time and resources required to train a model from scratch. These models support common tasks in different modalities, such as:

📝 **Natural Language Processing**: text classification, named entity recognition, question answering, language modeling, summarization, translation, multiple choice, and text generation.<br>
🖼️ **Computer Vision**: image classification, object detection, and segmentation.<br>
🗣️ **Audio**: automatic speech recognition and audio classification.<br>
🐙 **Multimodal**: table question answering, optical character recognition, information extraction from scanned documents, video classification, and visual question answering.

🤗 Transformers support framework interoperability between PyTorch, TensorFlow, and JAX. This provides the flexibility to use a different framework at each stage of a model's life; train a model in three lines of code in one framework, and load it for inference in another. Models can also be exported to a format like ONNX and TorchScript for deployment in production environments.

Join the growing community on the [Hub](https://huggingface.co/models), [forum](https://discuss.huggingface.co/), or [Discord](https://discord.com/invite/JfAtkvEtRb) today!

## If you are looking for custom support from the Hugging Face team

<a target="_blank" href="https://huggingface.co/support">
    <img alt="HuggingFace Expert Acceleration Program" src="https://cdn-media.huggingface.co/marketing/transformers/new-support-improved.png" style="width: 100%; max-width: 600px; border: 1px solid #eee; border-radius: 4px; box-shadow: 0 1px 2px 0 rgba(0, 0, 0, 0.05);">
</a>

## Contents

The documentation is organized into five sections:

- **GET STARTED** provides a quick tour of the library and installation instructions to get up and running.
- **TUTORIALS** are a great place to start if you're a beginner. This section will help you gain the basic skills you need to start using the library.
- **HOW-TO GUIDES** show you how to achieve a specific goal, like finetuning a pretrained model for language modeling or how to write and share a custom model.
- **CONCEPTUAL GUIDES** offers more discussion and explanation of the underlying concepts and ideas behind models, tasks, and the design philosophy of 🤗 Transformers.
- **API** describes all classes and functions:

  - **MAIN CLASSES** details the most important classes like configuration, model, tokenizer, and pipeline.
  - **MODELS** details the classes and functions related to each model implemented in the library.
  - **INTERNAL HELPERS** details utility classes and functions used internally.


## Supported models and frameworks

The table below represents the current support in the library for each of those models, whether they have a Python
tokenizer (called "slow"). A "fast" tokenizer backed by the 🤗 Tokenizers library, whether they have support in Jax (via
Flax), PyTorch, and/or TensorFlow.

<!--This table is updated automatically from the auto modules with _make fix-copies_. Do not update manually!-->

|


key: _node_content
string_value:
{"id_": "413c3208-e79b-49bf-9253-e391fd1280b2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "30e4aa16-77cb-4efb-926f-a05ec5bcb73e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "1a88f4d3ebfda474b1dcd6e67cd2573c21778fe7b4e724135f5e1afabb9d507b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "defe35de-8a6a-4060-b625-f582dc3a91a9", "node_type": "1", "metadata": {}, "hash": "f27304428015a00f0abdcb613b12ffafc4fff0501c11620868838b817f9331ba", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2980, "end_char_idx": 4432, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Supported models and frameworks

The table below represents the current support in the library for each of those models, whether they have a Python
tokenizer (called "slow"). A "fast" tokenizer backed by the 🤗 Tokenizers library, whether they have support in Jax (via
Flax), PyTorch, and/or TensorFlow.

<!--This table is updated automatically from the auto modules with _make fix-copies_. Do not update manually!-->

|                                  Model                                   | PyTorch support | TensorFlow support | Flax Support |
|:------------------------------------------------------------------------:|:---------------:|:------------------:|:------------:|
|                        [ALBERT](model_doc/albert)                        |       ✅        |         ✅         |      ✅      |
|                         [ALIGN](model_doc/align)                         |       ✅        |         ❌         |      ❌      |
|                       [AltCLIP](model_doc/altclip)                       |       ✅        |         ❌         |      ❌      |
| [Audio Spectrogram Transformer](model_doc/audio-spectrogram-transformer) |       ✅        |         ❌         |      ❌      |
|                    [Autoformer](model_doc/autoformer)                    |       ✅        |         ❌         |      ❌      |
|                          [Bark](model_doc/bark)                          |       ✅        |         ❌         |      ❌      |
|


key: _node_content
string_value:
{"id_": "defe35de-8a6a-4060-b625-f582dc3a91a9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "413c3208-e79b-49bf-9253-e391fd1280b2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "8ce74e2578721fb713d916aa1c23259f5a59ff2171dd3e62b492dac6d526c189", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5577f61f-0605-43b9-8356-2107aa67b6ba", "node_type": "1", "metadata": {}, "hash": "4a4e2d17774019d9d6b8b094a0602bb8f9a3b4c573f0fdb6db3c149072e06daf", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3903, "end_char_idx": 4899, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
|      ❌      |
|                       [AltCLIP](model_doc/altclip)                       |       ✅        |         ❌         |      ❌      |
| [Audio Spectrogram Transformer](model_doc/audio-spectrogram-transformer) |       ✅        |         ❌         |      ❌      |
|                    [Autoformer](model_doc/autoformer)                    |       ✅        |         ❌         |      ❌      |
|                          [Bark](model_doc/bark)                          |       ✅        |         ❌         |      ❌      |
|                          [BART](model_doc/bart)                          |       ✅        |         ✅         |      ✅      |
|                       [BARThez](model_doc/barthez)                       |       ✅        |         ✅         |      ✅      |
|                       [BARTpho](model_doc/bartpho)                       |       ✅        |         ✅         |      ✅      |
|                          [BEiT](model_doc/beit)                          |       ✅


key: _node_content
string_value:
{"id_": "5577f61f-0605-43b9-8356-2107aa67b6ba", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "defe35de-8a6a-4060-b625-f582dc3a91a9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "c678dbf601eeab09e98c185daba2cf7305046b194bf3fdc435a3453f05686c1d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f607a085-a865-4551-8544-9266a0851ce2", "node_type": "1", "metadata": {}, "hash": "4aab148442de04f6fce0fd7e9ca3c6dea2a5967a2571c2dc3da598c9751dbb4c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4422, "end_char_idx": 5403, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
❌      |
|                          [BART](model_doc/bart)                          |       ✅        |         ✅         |      ✅      |
|                       [BARThez](model_doc/barthez)                       |       ✅        |         ✅         |      ✅      |
|                       [BARTpho](model_doc/bartpho)                       |       ✅        |         ✅         |      ✅      |
|                          [BEiT](model_doc/beit)                          |       ✅        |         ❌         |      ✅      |
|                          [BERT](model_doc/bert)                          |       ✅        |         ✅         |      ✅      |
|               [Bert Generation](model_doc/bert-generation)               |       ✅        |         ❌         |      ❌      |
|                 [BertJapanese](model_doc/bert-japanese)                  |       ✅        |         ✅         |      ✅      |
|                      [BERTweet](model_doc/bertweet)                      |


key: _node_content
string_value:
{"id_": "f607a085-a865-4551-8544-9266a0851ce2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5577f61f-0605-43b9-8356-2107aa67b6ba", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "a1f6bf741a73fcdc428f3d6feb376b82f442c72a4ee085ff096f50e3a262e936", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8a46c85c-2e44-4399-ba12-b1bde9fe7a99", "node_type": "1", "metadata": {}, "hash": "5c3e2363b670e447da6b96644ff5425e5fccfea6058b06611b99cff3a31a68d3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4890, "end_char_idx": 5887, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
|       ✅        |         ❌         |      ✅      |
|                          [BERT](model_doc/bert)                          |       ✅        |         ✅         |      ✅      |
|               [Bert Generation](model_doc/bert-generation)               |       ✅        |         ❌         |      ❌      |
|                 [BertJapanese](model_doc/bert-japanese)                  |       ✅        |         ✅         |      ✅      |
|                      [BERTweet](model_doc/bertweet)                      |       ✅        |         ✅         |      ✅      |
|                      [BigBird](model_doc/big_bird)                       |       ✅        |         ❌         |      ✅      |
|               [BigBird-Pegasus](model_doc/bigbird_pegasus)               |       ✅        |         ❌         |      ❌      |
|                        [BioGpt](model_doc/biogpt)                        |       ✅        |         ❌         |      ❌      |
|                           [BiT](model_doc/bit)


key: _node_content
string_value:
{"id_": "8a46c85c-2e44-4399-ba12-b1bde9fe7a99", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f607a085-a865-4551-8544-9266a0851ce2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "27990e438d3790491785f00076a18e2d44ec3071bc47f2a9613762f503283752", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "345ac1a9-1ee5-4bb7-a65b-f6e1dd8aae11", "node_type": "1", "metadata": {}, "hash": "71aa929b084f1a39d702979d6228d3879478f78d69364613be36c4075c4b455e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5402, "end_char_idx": 6352, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
|       ✅        |         ✅         |      ✅      |
|                      [BigBird](model_doc/big_bird)                       |       ✅        |         ❌         |      ✅      |
|               [BigBird-Pegasus](model_doc/bigbird_pegasus)               |       ✅        |         ❌         |      ❌      |
|                        [BioGpt](model_doc/biogpt)                        |       ✅        |         ❌         |      ❌      |
|                           [BiT](model_doc/bit)                           |       ✅        |         ❌         |      ❌      |
|                    [Blenderbot](model_doc/blenderbot)                    |       ✅        |         ✅         |      ✅      |
|              [BlenderbotSmall](model_doc/blenderbot-small)               |       ✅        |         ✅         |      ✅      |
|                          [BLIP](model_doc/blip)                          |       ✅        |         ✅         |      ❌      |
|


key: _node_content
string_value:
{"id_": "345ac1a9-1ee5-4bb7-a65b-f6e1dd8aae11", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8a46c85c-2e44-4399-ba12-b1bde9fe7a99", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "a39d501ed65d14e062394b747b219912e1963d1ac3e31399da016b463c671003", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d327ea9f-df30-4da6-a438-464c42acf2dc", "node_type": "1", "metadata": {}, "hash": "e74e6a1e66f5c2bc11e7005ddf2136034bdac6165c77c25da7a54664c7241e24", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5867, "end_char_idx": 6838, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[BiT](model_doc/bit)                           |       ✅        |         ❌         |      ❌      |
|                    [Blenderbot](model_doc/blenderbot)                    |       ✅        |         ✅         |      ✅      |
|              [BlenderbotSmall](model_doc/blenderbot-small)               |       ✅        |         ✅         |      ✅      |
|                          [BLIP](model_doc/blip)                          |       ✅        |         ✅         |      ❌      |
|                        [BLIP-2](model_doc/blip-2)                        |       ✅        |         ❌         |      ❌      |
|                         [BLOOM](model_doc/bloom)                         |       ✅        |         ❌         |      ✅      |
|                          [BORT](model_doc/bort)                          |       ✅        |         ✅         |      ✅      |
|                   [BridgeTower](model_doc/bridgetower)                   |       ✅        |         ❌


key: _node_content
string_value:
{"id_": "d327ea9f-df30-4da6-a438-464c42acf2dc", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "345ac1a9-1ee5-4bb7-a65b-f6e1dd8aae11", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "bbf3e5f65346b3ab61cb38aea5634d2a8b0a46d51f0c63abce8b7c5b07cafa50", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "62525ed4-a424-4010-8674-f747babb45a6", "node_type": "1", "metadata": {}, "hash": "b2dd56751727902fa53c459d46c7a889fdf5b4c16c62fc0c4f83c4879b7b88d2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6376, "end_char_idx": 7298, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[BLIP-2](model_doc/blip-2)                        |       ✅        |         ❌         |      ❌      |
|                         [BLOOM](model_doc/bloom)                         |       ✅        |         ❌         |      ✅      |
|                          [BORT](model_doc/bort)                          |       ✅        |         ✅         |      ✅      |
|                   [BridgeTower](model_doc/bridgetower)                   |       ✅        |         ❌         |      ❌      |
|                          [BROS](model_doc/bros)                          |       ✅        |         ❌         |      ❌      |
|                          [ByT5](model_doc/byt5)                          |       ✅        |         ✅         |      ✅      |
|                     [CamemBERT](model_doc/camembert)                     |       ✅        |         ✅         |      ❌      |
|                        [CANINE](model_doc/canine)


key: _node_content
string_value:
{"id_": "62525ed4-a424-4010-8674-f747babb45a6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d327ea9f-df30-4da6-a438-464c42acf2dc", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "e2b33adaf81ead8291cc018116c843c44270d2949e34847223c14fb3cb878274", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "57f61a2c-0140-4297-85ef-b90e0f80fbed", "node_type": "1", "metadata": {}, "hash": "4cbf91152004a90e036da288176f074f384967efb5baa74fdcab13d3ec8f5d25", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6827, "end_char_idx": 7760, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
|         ❌         |      ❌      |
|                          [BROS](model_doc/bros)                          |       ✅        |         ❌         |      ❌      |
|                          [ByT5](model_doc/byt5)                          |       ✅        |         ✅         |      ✅      |
|                     [CamemBERT](model_doc/camembert)                     |       ✅        |         ✅         |      ❌      |
|                        [CANINE](model_doc/canine)                        |       ✅        |         ❌         |      ❌      |
|                     [Chameleon](model_doc/chameleon)                     |       ✅        |         ❌         |      ❌      |
|                  [Chinese-CLIP](model_doc/chinese_clip)                  |       ✅        |         ❌         |      ❌      |
|                          [CLAP](model_doc/clap)                          |       ✅        |         ❌         |      ❌      |
|


key: _node_content
string_value:
{"id_": "57f61a2c-0140-4297-85ef-b90e0f80fbed", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "62525ed4-a424-4010-8674-f747babb45a6", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "575d3338705bd3b29f247dd3f5136a03e3a1c7488139a0796b1dc8827040ebfd", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e8028b6c-10d9-4e9b-bbe2-a5b8beb06875", "node_type": "1", "metadata": {}, "hash": "4a0f4abfcf30f08b776cedba1cd38b68ecd5c270266a7a1498278a3dba20ee8e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7322, "end_char_idx": 8246, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
|       ✅        |         ❌         |      ❌      |
|                     [Chameleon](model_doc/chameleon)                     |       ✅        |         ❌         |      ❌      |
|                  [Chinese-CLIP](model_doc/chinese_clip)                  |       ✅        |         ❌         |      ❌      |
|                          [CLAP](model_doc/clap)                          |       ✅        |         ❌         |      ❌      |
|                          [CLIP](model_doc/clip)                          |       ✅        |         ✅         |      ✅      |
|                       [CLIPSeg](model_doc/clipseg)                       |       ✅        |         ❌         |      ❌      |
|                          [CLVP](model_doc/clvp)                          |       ✅        |         ❌         |      ❌      |
|                       [CodeGen](model_doc/codegen)                       |       ✅        |         ❌


key: _node_content
string_value:
{"id_": "e8028b6c-10d9-4e9b-bbe2-a5b8beb06875", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "57f61a2c-0140-4297-85ef-b90e0f80fbed", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "1d16c36921c76fa9c81ccd95334deacbac506b7239d0ea854e9da20355f7a5b7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a4c43eb1-d01f-45e7-98a1-1bdfff39ae5c", "node_type": "1", "metadata": {}, "hash": "9178f9e3f21e46b64e96628c8067bbac6d78b7dfa7bb76712a919c48a2d1f5c1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7786, "end_char_idx": 8748, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[CLIP](model_doc/clip)                          |       ✅        |         ✅         |      ✅      |
|                       [CLIPSeg](model_doc/clipseg)                       |       ✅        |         ❌         |      ❌      |
|                          [CLVP](model_doc/clvp)                          |       ✅        |         ❌         |      ❌      |
|                       [CodeGen](model_doc/codegen)                       |       ✅        |         ❌         |      ❌      |
|                    [CodeLlama](model_doc/code_llama)                     |       ✅        |         ❌         |      ✅      |
|                        [Cohere](model_doc/cohere)                        |       ✅        |         ❌         |      ❌      |
|              [Conditional DETR](model_doc/conditional_detr)              |       ✅        |         ❌         |      ❌      |
|                      [ConvBERT](model_doc/convbert)                      |       ✅        |


key: _node_content
string_value:
{"id_": "a4c43eb1-d01f-45e7-98a1-1bdfff39ae5c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e8028b6c-10d9-4e9b-bbe2-a5b8beb06875", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "32a9d7a97571bf13b55101ab59c1913c10236644635f46b7b3470fed8e61c345", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2b629d82-c660-4c8f-b041-761073811472", "node_type": "1", "metadata": {}, "hash": "2a4f7c252c21c51954fc0b18ec19f6e3295db3965d9d2e47219bdb643c984fa2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8245, "end_char_idx": 9218, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
❌         |      ❌      |
|                    [CodeLlama](model_doc/code_llama)                     |       ✅        |         ❌         |      ✅      |
|                        [Cohere](model_doc/cohere)                        |       ✅        |         ❌         |      ❌      |
|              [Conditional DETR](model_doc/conditional_detr)              |       ✅        |         ❌         |      ❌      |
|                      [ConvBERT](model_doc/convbert)                      |       ✅        |         ✅         |      ❌      |
|                      [ConvNeXT](model_doc/convnext)                      |       ✅        |         ✅         |      ❌      |
|                    [ConvNeXTV2](model_doc/convnextv2)                    |       ✅        |         ✅         |      ❌      |
|                           [CPM](model_doc/cpm)                           |       ✅        |         ✅         |      ✅      |
|                       [CPM-Ant](model_doc/cpmant)


key: _node_content
string_value:
{"id_": "2b629d82-c660-4c8f-b041-761073811472", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a4c43eb1-d01f-45e7-98a1-1bdfff39ae5c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "a6d572eca60573af9674354fcfb2c07b2fa999537f193653c40c4c0c6030bba3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4d1f5583-5a15-45a9-895e-12c03b3f4b69", "node_type": "1", "metadata": {}, "hash": "72eb6b26eb75a5feb7f75b7cef29a99a68b733446b70a8e014653c2351b54429", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8747, "end_char_idx": 9671, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
|         ✅         |      ❌      |
|                      [ConvNeXT](model_doc/convnext)                      |       ✅        |         ✅         |      ❌      |
|                    [ConvNeXTV2](model_doc/convnextv2)                    |       ✅        |         ✅         |      ❌      |
|                           [CPM](model_doc/cpm)                           |       ✅        |         ✅         |      ✅      |
|                       [CPM-Ant](model_doc/cpmant)                        |       ✅        |         ❌         |      ❌      |
|                          [CTRL](model_doc/ctrl)                          |       ✅        |         ✅         |      ❌      |
|                           [CvT](model_doc/cvt)                           |       ✅        |         ✅         |      ❌      |
|                           [DAC](model_doc/dac)                           |       ✅        |         ❌         |      ❌


key: _node_content
string_value:
{"id_": "4d1f5583-5a15-45a9-895e-12c03b3f4b69", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2b629d82-c660-4c8f-b041-761073811472", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "61c3ddfe0a41b2171fee2933d9f3624fd2be3a51989779c7f422767bf53a571b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "cd69eec9-7fe1-4f78-8066-cb71fa360a91", "node_type": "1", "metadata": {}, "hash": "146338ee34189f094dba32544f9794d74be6a2d87b78e1895fb7f8e56c4467b4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9242, "end_char_idx": 10166, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
|       ✅        |         ❌         |      ❌      |
|                          [CTRL](model_doc/ctrl)                          |       ✅        |         ✅         |      ❌      |
|                           [CvT](model_doc/cvt)                           |       ✅        |         ✅         |      ❌      |
|                           [DAC](model_doc/dac)                           |       ✅        |         ❌         |      ❌      |
|                   [Data2VecAudio](model_doc/data2vec)                    |       ✅        |         ❌         |      ❌      |
|                    [Data2VecText](model_doc/data2vec)                    |       ✅        |         ❌         |      ❌      |
|                   [Data2VecVision](model_doc/data2vec)                   |       ✅        |         ✅         |      ❌      |
|                          [DBRX](model_doc/dbrx)                          |       ✅        |         ❌


key: _node_content
string_value:
{"id_": "cd69eec9-7fe1-4f78-8066-cb71fa360a91", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4d1f5583-5a15-45a9-895e-12c03b3f4b69", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "46c08edcf8b3b6bbc7fc1e7b9e61e1e939b1f0ca7da0b57d54aee09745b5a287", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a58bd46b-7ddc-490c-bc56-f9a13c9a640a", "node_type": "1", "metadata": {}, "hash": "25b357628f290774fda03ef2c2757843387ef442727c4a691a54e7603757171a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9663, "end_char_idx": 10678, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
|      ❌      |
|                   [Data2VecAudio](model_doc/data2vec)                    |       ✅        |         ❌         |      ❌      |
|                    [Data2VecText](model_doc/data2vec)                    |       ✅        |         ❌         |      ❌      |
|                   [Data2VecVision](model_doc/data2vec)                   |       ✅        |         ✅         |      ❌      |
|                          [DBRX](model_doc/dbrx)                          |       ✅        |         ❌         |      ❌      |
|                       [DeBERTa](model_doc/deberta)                       |       ✅        |         ✅         |      ❌      |
|                    [DeBERTa-v2](model_doc/deberta-v2)                    |       ✅        |         ✅         |      ❌      |
|          [Decision Transformer](model_doc/decision_transformer)          |       ✅        |         ❌         |      ❌      |
|               [Deformable DETR](model_doc/deformable_detr)               |       ✅        |         ❌


key: _node_content
string_value:
{"id_": "a58bd46b-7ddc-490c-bc56-f9a13c9a640a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "cd69eec9-7fe1-4f78-8066-cb71fa360a91", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "ab8cb4b3690123e733282f66515113f23ac7b8383f9611473dfd3d8287cb8988", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9562b85b-a1f4-4193-8ed7-f977a5358b32", "node_type": "1", "metadata": {}, "hash": "e479cbe608b86ceee5abc8877dcd4dbd55c265fadb0eefbee6411085ea38ac74", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10155, "end_char_idx": 11163, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
|         ❌         |      ❌      |
|                       [DeBERTa](model_doc/deberta)                       |       ✅        |         ✅         |      ❌      |
|                    [DeBERTa-v2](model_doc/deberta-v2)                    |       ✅        |         ✅         |      ❌      |
|          [Decision Transformer](model_doc/decision_transformer)          |       ✅        |         ❌         |      ❌      |
|               [Deformable DETR](model_doc/deformable_detr)               |       ✅        |         ❌         |      ❌      |
|                          [DeiT](model_doc/deit)                          |       ✅        |         ✅         |      ❌      |
|                        [DePlot](model_doc/deplot)                        |       ✅        |         ❌         |      ❌      |
|                [Depth Anything](model_doc/depth_anything)                |       ✅        |         ❌         |      ❌      |
|                          [DETA](model_doc/deta)                          |


key: _node_content
string_value:
{"id_": "9562b85b-a1f4-4193-8ed7-f977a5358b32", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a58bd46b-7ddc-490c-bc56-f9a13c9a640a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "15b7cb362054a2759c1c0d32016ad0c0bc5d085ef01cfe8d334117cecf386b03", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2408f6a0-a66f-4bda-bd44-0a2135a0a8b9", "node_type": "1", "metadata": {}, "hash": "b34a36b2f72f6edba77d75b6ba80047f9bc0b69cc024728a1da61013117f4d4c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10677, "end_char_idx": 11600, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
❌         |      ❌      |
|                          [DeiT](model_doc/deit)                          |       ✅        |         ✅         |      ❌      |
|                        [DePlot](model_doc/deplot)                        |       ✅        |         ❌         |      ❌      |
|                [Depth Anything](model_doc/depth_anything)                |       ✅        |         ❌         |      ❌      |
|                          [DETA](model_doc/deta)                          |       ✅        |         ❌         |      ❌      |
|                          [DETR](model_doc/detr)                          |       ✅        |         ❌         |      ❌      |
|                      [DialoGPT](model_doc/dialogpt)                      |       ✅        |         ✅         |      ✅      |
|                         [DiNAT](model_doc/dinat)                         |       ✅        |         ❌         |      ❌      |
|


key: _node_content
string_value:
{"id_": "2408f6a0-a66f-4bda-bd44-0a2135a0a8b9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9562b85b-a1f4-4193-8ed7-f977a5358b32", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "ebd432f1f04f63ea76c8f999b5385f9c43f4392db23414a3baee044a9616d9ba", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d05afd92-2be2-430f-afdd-4c2df68abd2d", "node_type": "1", "metadata": {}, "hash": "f956cd909d2987f06c57eca4c02f46f0a659eb6fe69e5f0f951473447c6208b4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11162, "end_char_idx": 12086, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
|       ✅        |         ❌         |      ❌      |
|                          [DETR](model_doc/detr)                          |       ✅        |         ❌         |      ❌      |
|                      [DialoGPT](model_doc/dialogpt)                      |       ✅        |         ✅         |      ✅      |
|                         [DiNAT](model_doc/dinat)                         |       ✅        |         ❌         |      ❌      |
|                        [DINOv2](model_doc/dinov2)                        |       ✅        |         ❌         |      ✅      |
|                    [DistilBERT](model_doc/distilbert)                    |       ✅        |         ✅         |      ✅      |
|                           [DiT](model_doc/dit)                           |       ✅        |         ❌         |      ✅      |
|                       [DonutSwin](model_doc/donut)                       |       ✅        |         ❌


key: _node_content
string_value:
{"id_": "d05afd92-2be2-430f-afdd-4c2df68abd2d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2408f6a0-a66f-4bda-bd44-0a2135a0a8b9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "0b56b37bc44b9b15a2b2a463a1935a667e54356a89c1ffc410e8b75e372ff06e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1530ca5d-f76a-432c-aeba-70ca14e0984b", "node_type": "1", "metadata": {}, "hash": "1cf2701173afdda54b152b6bfe514461d89b93984fe6955fac19a3accbfea78f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11624, "end_char_idx": 12579, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[DINOv2](model_doc/dinov2)                        |       ✅        |         ❌         |      ✅      |
|                    [DistilBERT](model_doc/distilbert)                    |       ✅        |         ✅         |      ✅      |
|                           [DiT](model_doc/dit)                           |       ✅        |         ❌         |      ✅      |
|                       [DonutSwin](model_doc/donut)                       |       ✅        |         ❌         |      ❌      |
|                           [DPR](model_doc/dpr)                           |       ✅        |         ✅         |      ❌      |
|                           [DPT](model_doc/dpt)                           |       ✅        |         ❌         |      ❌      |
|               [EfficientFormer](model_doc/efficientformer)               |       ✅        |         ✅         |      ❌      |
|                  [EfficientNet](model_doc/efficientnet)                  |       ✅


key: _node_content
string_value:
{"id_": "1530ca5d-f76a-432c-aeba-70ca14e0984b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d05afd92-2be2-430f-afdd-4c2df68abd2d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "7754e7dcd12d86863e8f5b126243fb71142085e96d0954c4170419ff998cabde", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5f84b30a-3af8-486e-abb7-122c782397ee", "node_type": "1", "metadata": {}, "hash": "b43490eb9793c6a2b8479c961ec8b544493367769508baab685017e58905ab80", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12085, "end_char_idx": 13057, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
❌         |      ❌      |
|                           [DPR](model_doc/dpr)                           |       ✅        |         ✅         |      ❌      |
|                           [DPT](model_doc/dpt)                           |       ✅        |         ❌         |      ❌      |
|               [EfficientFormer](model_doc/efficientformer)               |       ✅        |         ✅         |      ❌      |
|                  [EfficientNet](model_doc/efficientnet)                  |       ✅        |         ❌         |      ❌      |
|                       [ELECTRA](model_doc/electra)                       |       ✅        |         ✅         |      ✅      |
|                       [EnCodec](model_doc/encodec)                       |       ✅        |         ❌         |      ❌      |
|               [Encoder decoder](model_doc/encoder-decoder)               |       ✅        |         ✅         |      ✅      |
|                         [ERNIE](model_doc/ernie)


key: _node_content
string_value:
{"id_": "5f84b30a-3af8-486e-abb7-122c782397ee", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1530ca5d-f76a-432c-aeba-70ca14e0984b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "dfb48eeaafd3b2ff300e21d75cf1b5a96fc940b20e8d5102f619eb7515ba2a0d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "783093db-90d4-4c57-86f8-2c217075e83a", "node_type": "1", "metadata": {}, "hash": "ba06dfc450c8a640406eaadec0d5acab416ac291290f559eadf66ed9b71010f9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12578, "end_char_idx": 13520, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
✅        |         ❌         |      ❌      |
|                       [ELECTRA](model_doc/electra)                       |       ✅        |         ✅         |      ✅      |
|                       [EnCodec](model_doc/encodec)                       |       ✅        |         ❌         |      ❌      |
|               [Encoder decoder](model_doc/encoder-decoder)               |       ✅        |         ✅         |      ✅      |
|                         [ERNIE](model_doc/ernie)                         |       ✅        |         ❌         |      ❌      |
|                       [ErnieM](model_doc/ernie_m)                        |       ✅        |         ❌         |      ❌      |
|                           [ESM](model_doc/esm)                           |       ✅        |         ✅         |      ❌      |
|              [FairSeq Machine-Translation](model_doc/fsmt)               |       ✅        |         ❌         |      ❌      |
|


key: _node_content
string_value:
{"id_": "783093db-90d4-4c57-86f8-2c217075e83a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5f84b30a-3af8-486e-abb7-122c782397ee", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "6893049ea68f2c689f2ed1748cec6cc5b176b6d20242e663362c6d030ad28b53", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f7575cca-9bc3-4a02-b253-0a8f67a53355", "node_type": "1", "metadata": {}, "hash": "817a83d55e6cf71248d8786a8e5dedc38a3e2ed02756b5b9c67e1026a39c216f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13082, "end_char_idx": 14032, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
|       ✅        |         ❌         |      ❌      |
|                       [ErnieM](model_doc/ernie_m)                        |       ✅        |         ❌         |      ❌      |
|                           [ESM](model_doc/esm)                           |       ✅        |         ✅         |      ❌      |
|              [FairSeq Machine-Translation](model_doc/fsmt)               |       ✅        |         ❌         |      ❌      |
|                        [Falcon](model_doc/falcon)                        |       ✅        |         ❌         |      ❌      |
|                  [FalconMamba](model_doc/falcon_mamba)                   |       ✅        |         ❌         |      ❌      |
|         [FastSpeech2Conformer](model_doc/fastspeech2_conformer)          |       ✅        |         ❌         |      ❌      |
|                       [FLAN-T5](model_doc/flan-t5)                       |       ✅        |         ✅         |      ✅      |
|


key: _node_content
string_value:
{"id_": "f7575cca-9bc3-4a02-b253-0a8f67a53355", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "783093db-90d4-4c57-86f8-2c217075e83a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "bdad54499e8229c2cef66535f32ebfdc292898eb130b8f1f458abab950e37545", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "cefba0f5-acec-4509-940e-76746d2557a2", "node_type": "1", "metadata": {}, "hash": "6722b6f60bd608143ba7b00a077a120040f079ce8ab17f595544dfcb947514dc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13544, "end_char_idx": 14518, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[Falcon](model_doc/falcon)                        |       ✅        |         ❌         |      ❌      |
|                  [FalconMamba](model_doc/falcon_mamba)                   |       ✅        |         ❌         |      ❌      |
|         [FastSpeech2Conformer](model_doc/fastspeech2_conformer)          |       ✅        |         ❌         |      ❌      |
|                       [FLAN-T5](model_doc/flan-t5)                       |       ✅        |         ✅         |      ✅      |
|                      [FLAN-UL2](model_doc/flan-ul2)                      |       ✅        |         ✅         |      ✅      |
|                      [FlauBERT](model_doc/flaubert)                      |       ✅        |         ✅         |      ❌      |
|                         [FLAVA](model_doc/flava)                         |       ✅        |         ❌         |      ❌      |
|                          [FNet](model_doc/fnet)                          |       ✅        |         ❌


key: _node_content
string_value:
{"id_": "cefba0f5-acec-4509-940e-76746d2557a2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f7575cca-9bc3-4a02-b253-0a8f67a53355", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "a10a13be1f29d6735da52fc98a37372866c40a032d3811e499e92a6e2b1a5ac9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7a6235a6-399d-4107-b004-9f5f7c26a0c4", "node_type": "1", "metadata": {}, "hash": "c62dacd6c8334f153fb423e48d39febdda9d69a426f37caf2c644385f2997d3d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14054, "end_char_idx": 15003, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[FLAN-UL2](model_doc/flan-ul2)                      |       ✅        |         ✅         |      ✅      |
|                      [FlauBERT](model_doc/flaubert)                      |       ✅        |         ✅         |      ❌      |
|                         [FLAVA](model_doc/flava)                         |       ✅        |         ❌         |      ❌      |
|                          [FNet](model_doc/fnet)                          |       ✅        |         ❌         |      ❌      |
|                      [FocalNet](model_doc/focalnet)                      |       ✅        |         ❌         |      ❌      |
|                  [Funnel Transformer](model_doc/funnel)                  |       ✅        |         ✅         |      ❌      |
|                          [Fuyu](model_doc/fuyu)                          |       ✅        |         ❌         |      ❌      |
|                         [Gemma](model_doc/gemma)                         |


key: _node_content
string_value:
{"id_": "7a6235a6-399d-4107-b004-9f5f7c26a0c4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "cefba0f5-acec-4509-940e-76746d2557a2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "9ff1d72d0588a6d949748d3414cc26ffabe4127e978b89f352dbab9530b53c8b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9395be22-9e57-4bfd-9d7a-6430b1b4f6c9", "node_type": "1", "metadata": {}, "hash": "835a3d5f40d2c71d06daa7164287f651afdeccc8477cec9b8fc3699dd59493ab", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14507, "end_char_idx": 15440, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
|         ❌         |      ❌      |
|                      [FocalNet](model_doc/focalnet)                      |       ✅        |         ❌         |      ❌      |
|                  [Funnel Transformer](model_doc/funnel)                  |       ✅        |         ✅         |      ❌      |
|                          [Fuyu](model_doc/fuyu)                          |       ✅        |         ❌         |      ❌      |
|                         [Gemma](model_doc/gemma)                         |       ✅        |         ❌         |      ✅      |
|                        [Gemma2](model_doc/gemma2)                        |       ✅        |         ❌         |      ❌      |
|                           [GIT](model_doc/git)                           |       ✅        |         ❌         |      ❌      |
|                          [GLPN](model_doc/glpn)                          |       ✅        |         ❌         |      ❌      |
|


key: _node_content
string_value:
{"id_": "9395be22-9e57-4bfd-9d7a-6430b1b4f6c9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7a6235a6-399d-4107-b004-9f5f7c26a0c4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "9678bd93c72034a6dd8f26c67835c41bfe321727a261b24eb76c2c59b3a7ae45", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "675ebdfb-ea3d-4298-a176-65063e0c6128", "node_type": "1", "metadata": {}, "hash": "0e09258f755c5221efb430967efc0525bdf0ec1a8aa0d998d853780c17b6f93a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15002, "end_char_idx": 15936, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
|       ✅        |         ❌         |      ✅      |
|                        [Gemma2](model_doc/gemma2)                        |       ✅        |         ❌         |      ❌      |
|                           [GIT](model_doc/git)                           |       ✅        |         ❌         |      ❌      |
|                          [GLPN](model_doc/glpn)                          |       ✅        |         ❌         |      ❌      |
|                       [GPT Neo](model_doc/gpt_neo)                       |       ✅        |         ❌         |      ✅      |
|                      [GPT NeoX](model_doc/gpt_neox)                      |       ✅        |         ❌         |      ❌      |
|             [GPT NeoX Japanese](model_doc/gpt_neox_japanese)             |       ✅        |         ❌         |      ❌      |
|                         [GPT-J](model_doc/gptj)                          |       ✅        |         ✅         |


key: _node_content
string_value:
{"id_": "675ebdfb-ea3d-4298-a176-65063e0c6128", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9395be22-9e57-4bfd-9d7a-6430b1b4f6c9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "b1dc8c0f7c352e8facade29e11c6f9fad44aa07f8025a0ac59658057f21a8cee", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5930b80b-750e-441d-a0b0-016a85c0c998", "node_type": "1", "metadata": {}, "hash": "818de82d31a5db6ee0f451db70bcaab7e444762fcfffc4c24c0c200043d328cc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15463, "end_char_idx": 16428, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[GPT Neo](model_doc/gpt_neo)                       |       ✅        |         ❌         |      ✅      |
|                      [GPT NeoX](model_doc/gpt_neox)                      |       ✅        |         ❌         |      ❌      |
|             [GPT NeoX Japanese](model_doc/gpt_neox_japanese)             |       ✅        |         ❌         |      ❌      |
|                         [GPT-J](model_doc/gptj)                          |       ✅        |         ✅         |      ✅      |
|                       [GPT-Sw3](model_doc/gpt-sw3)                       |       ✅        |         ✅         |      ✅      |
|                   [GPTBigCode](model_doc/gpt_bigcode)                    |       ✅        |         ❌         |      ❌      |
|               [GPTSAN-japanese](model_doc/gptsan-japanese)               |       ✅        |         ❌         |      ❌      |
|                    [Graphormer](model_doc/graphormer)                    |       ✅        |


key: _node_content
string_value:
{"id_": "5930b80b-750e-441d-a0b0-016a85c0c998", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "675ebdfb-ea3d-4298-a176-65063e0c6128", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "b1026c11a801ffceb4c62d8b927841aa691f29058b283020dee0aa412c8e701e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "70127e61-3391-411a-b291-ac102a7c8ea9", "node_type": "1", "metadata": {}, "hash": "e17aa38f94dcc590d7ae267a025f502f2f0c505b50bf4bffcc007eb5dfbf5945", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15925, "end_char_idx": 16923, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
✅         |      ✅      |
|                       [GPT-Sw3](model_doc/gpt-sw3)                       |       ✅        |         ✅         |      ✅      |
|                   [GPTBigCode](model_doc/gpt_bigcode)                    |       ✅        |         ❌         |      ❌      |
|               [GPTSAN-japanese](model_doc/gptsan-japanese)               |       ✅        |         ❌         |      ❌      |
|                    [Graphormer](model_doc/graphormer)                    |       ✅        |         ❌         |      ❌      |
|                [Grounding DINO](model_doc/grounding-dino)                |       ✅        |         ❌         |      ❌      |
|                      [GroupViT](model_doc/groupvit)                      |       ✅        |         ✅         |      ❌      |
|                       [HerBERT](model_doc/herbert)                       |       ✅        |         ✅         |      ✅      |
|                         [Hiera](model_doc/hiera)                         |


key: _node_content
string_value:
{"id_": "70127e61-3391-411a-b291-ac102a7c8ea9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5930b80b-750e-441d-a0b0-016a85c0c998", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "2f262907e803416f3e202c9059e479e7a9d84783277bb50671adce421dd8152f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "030b90f8-15da-49e5-894a-34a2c422e5da", "node_type": "1", "metadata": {}, "hash": "4b3bb7991887925d62a521fe0c931874d4b63a0c5e8a89d7d6e9dc3f4949946c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16427, "end_char_idx": 17360, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
|         ❌         |      ❌      |
|                [Grounding DINO](model_doc/grounding-dino)                |       ✅        |         ❌         |      ❌      |
|                      [GroupViT](model_doc/groupvit)                      |       ✅        |         ✅         |      ❌      |
|                       [HerBERT](model_doc/herbert)                       |       ✅        |         ✅         |      ✅      |
|                         [Hiera](model_doc/hiera)                         |       ✅        |         ❌         |      ❌      |
|                        [Hubert](model_doc/hubert)                        |       ✅        |         ✅         |      ❌      |
|                        [I-BERT](model_doc/ibert)                         |       ✅        |         ❌         |      ❌      |
|                       [IDEFICS](model_doc/idefics)                       |       ✅        |         ✅         |      ❌      |
|


key: _node_content
string_value:
{"id_": "030b90f8-15da-49e5-894a-34a2c422e5da", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "70127e61-3391-411a-b291-ac102a7c8ea9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "d4c2ad91c20088b8e0041b48841635300a45765f8ddc7a45751d083613f390a8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8fada690-73db-40b7-ae97-f70f4e35ffb6", "node_type": "1", "metadata": {}, "hash": "168e1fbc1c198d78f3af1d5dde355130c099ec9c02af4d0a42299bbde74a21d5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16922, "end_char_idx": 17872, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
|       ✅        |         ❌         |      ❌      |
|                        [Hubert](model_doc/hubert)                        |       ✅        |         ✅         |      ❌      |
|                        [I-BERT](model_doc/ibert)                         |       ✅        |         ❌         |      ❌      |
|                       [IDEFICS](model_doc/idefics)                       |       ✅        |         ✅         |      ❌      |
|                      [Idefics2](model_doc/idefics2)                      |       ✅        |         ❌         |      ❌      |
|                      [ImageGPT](model_doc/imagegpt)                      |       ✅        |         ❌         |      ❌      |
|                      [Informer](model_doc/informer)                      |       ✅        |         ❌         |      ❌      |
|                  [InstructBLIP](model_doc/instructblip)                  |       ✅        |         ❌         |      ❌      |
|


key: _node_content
string_value:
{"id_": "8fada690-73db-40b7-ae97-f70f4e35ffb6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "030b90f8-15da-49e5-894a-34a2c422e5da", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "c66845cc1a1c13586768e0adf39a00a6d24e2eaa82b1293393db0f035e3b8732", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0c7c7cd6-9903-49db-b7d2-a035e103213d", "node_type": "1", "metadata": {}, "hash": "cee0910cdba09d4dca09ad00c67ffe8978056a3cbedd870fc81d9ec48b06f511", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17382, "end_char_idx": 18358, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[Idefics2](model_doc/idefics2)                      |       ✅        |         ❌         |      ❌      |
|                      [ImageGPT](model_doc/imagegpt)                      |       ✅        |         ❌         |      ❌      |
|                      [Informer](model_doc/informer)                      |       ✅        |         ❌         |      ❌      |
|                  [InstructBLIP](model_doc/instructblip)                  |       ✅        |         ❌         |      ❌      |
|             [InstructBlipVideo](model_doc/instructblipvideo)             |       ✅        |         ❌         |      ❌      |
|                         [Jamba](model_doc/jamba)                         |       ✅        |         ❌         |      ❌      |
|                        [JetMoe](model_doc/jetmoe)                        |       ✅        |         ❌         |      ❌      |
|                       [Jukebox](model_doc/jukebox)                       |       ✅        |         ❌


key: _node_content
string_value:
{"id_": "0c7c7cd6-9903-49db-b7d2-a035e103213d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8fada690-73db-40b7-ae97-f70f4e35ffb6", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "fd1cdc4c7aef5aeff4a27be14098782a5f77df8db91483a09165c9274fed9c01", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a8ad3174-1fcf-4964-9a25-a7989b36d1f3", "node_type": "1", "metadata": {}, "hash": "c56503a4c960489b8baa2c8efde72a975fb3c747058ee81e7a367929a9ccf331", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17862, "end_char_idx": 18843, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
❌      |
|             [InstructBlipVideo](model_doc/instructblipvideo)             |       ✅        |         ❌         |      ❌      |
|                         [Jamba](model_doc/jamba)                         |       ✅        |         ❌         |      ❌      |
|                        [JetMoe](model_doc/jetmoe)                        |       ✅        |         ❌         |      ❌      |
|                       [Jukebox](model_doc/jukebox)                       |       ✅        |         ❌         |      ❌      |
|                      [KOSMOS-2](model_doc/kosmos-2)                      |       ✅        |         ❌         |      ❌      |
|                      [LayoutLM](model_doc/layoutlm)                      |       ✅        |         ✅         |      ❌      |
|                    [LayoutLMv2](model_doc/layoutlmv2)                    |       ✅        |         ❌         |      ❌      |
|                    [LayoutLMv3](model_doc/layoutlmv3)                    |


key: _node_content
string_value:
{"id_": "a8ad3174-1fcf-4964-9a25-a7989b36d1f3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0c7c7cd6-9903-49db-b7d2-a035e103213d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "9cb615a86d1b1453e83dbc0373d5471294cccd45c14dd924380896ed5ada9044", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b557fca7-6b28-4e1f-8296-adda35e77b80", "node_type": "1", "metadata": {}, "hash": "077477572b7d118fd2dbb0e06b1ef0dd1f4676e1af0ad9f2ceb0c3a00fadabb0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18347, "end_char_idx": 19280, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
|         ❌         |      ❌      |
|                      [KOSMOS-2](model_doc/kosmos-2)                      |       ✅        |         ❌         |      ❌      |
|                      [LayoutLM](model_doc/layoutlm)                      |       ✅        |         ✅         |      ❌      |
|                    [LayoutLMv2](model_doc/layoutlmv2)                    |       ✅        |         ❌         |      ❌      |
|                    [LayoutLMv3](model_doc/layoutlmv3)                    |       ✅        |         ✅         |      ❌      |
|                     [LayoutXLM](model_doc/layoutxlm)                     |       ✅        |         ❌         |      ❌      |
|                           [LED](model_doc/led)                           |       ✅        |         ✅         |      ❌      |
|                         [LeViT](model_doc/levit)                         |       ✅        |         ❌         |      ❌      |
|


key: _node_content
string_value:
{"id_": "b557fca7-6b28-4e1f-8296-adda35e77b80", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a8ad3174-1fcf-4964-9a25-a7989b36d1f3", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "32ee90448fe6c075196db4cbbfd4bcc73876ee2c21039e00cddd020b8079f197", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "02f79f93-d2fe-42b0-b405-fb71e5a6546c", "node_type": "1", "metadata": {}, "hash": "79a7bc66d7a967e1989eaf55e4416e338ac89fba50c43b4b2b05ae7c3bd919fb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18842, "end_char_idx": 19756, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
|       ✅        |         ✅         |      ❌      |
|                     [LayoutXLM](model_doc/layoutxlm)                     |       ✅        |         ❌         |      ❌      |
|                           [LED](model_doc/led)                           |       ✅        |         ✅         |      ❌      |
|                         [LeViT](model_doc/levit)                         |       ✅        |         ❌         |      ❌      |
|                          [LiLT](model_doc/lilt)                          |       ✅        |         ❌         |      ❌      |
|                         [LLaMA](model_doc/llama)                         |       ✅        |         ❌         |      ✅      |
|                        [Llama2](model_doc/llama2)                        |       ✅        |         ❌         |      ✅      |
|                        [Llama3](model_doc/llama3)                        |       ✅        |


key: _node_content
string_value:
{"id_": "02f79f93-d2fe-42b0-b405-fb71e5a6546c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b557fca7-6b28-4e1f-8296-adda35e77b80", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "9a5d3973298b0c833c133539273900544ab5e795f415dcdeaf0bc463f04218ab", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "832b5863-6a7d-4fe5-889d-679f91578cf0", "node_type": "1", "metadata": {}, "hash": "737388c2865d3a4672c8d115536a30fa25554314553c4c3aed6eae1b3fda0a39", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19306, "end_char_idx": 20251, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[LiLT](model_doc/lilt)                          |       ✅        |         ❌         |      ❌      |
|                         [LLaMA](model_doc/llama)                         |       ✅        |         ❌         |      ✅      |
|                        [Llama2](model_doc/llama2)                        |       ✅        |         ❌         |      ✅      |
|                        [Llama3](model_doc/llama3)                        |       ✅        |         ❌         |      ✅      |
|                         [LLaVa](model_doc/llava)                         |       ✅        |         ❌         |      ❌      |
|                    [LLaVA-NeXT](model_doc/llava_next)                    |       ✅        |         ❌         |      ❌      |
|              [LLaVa-NeXT-Video](model_doc/llava_next_video)              |       ✅        |         ❌         |      ❌      |
|                    [Longformer](model_doc/longformer)                    |


key: _node_content
string_value:
{"id_": "832b5863-6a7d-4fe5-889d-679f91578cf0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "02f79f93-d2fe-42b0-b405-fb71e5a6546c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "566ba572ad21fdc43517b1c6a19c5711de05106242df3692993cba95dd93517a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "aaa980f0-e175-45ec-b693-d3abfd96e34f", "node_type": "1", "metadata": {}, "hash": "091e509c6aca40855c004a0826c1f60e14551b0bf28a6384773349f6cbe39e9b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19755, "end_char_idx": 20688, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
|         ❌         |      ✅      |
|                         [LLaVa](model_doc/llava)                         |       ✅        |         ❌         |      ❌      |
|                    [LLaVA-NeXT](model_doc/llava_next)                    |       ✅        |         ❌         |      ❌      |
|              [LLaVa-NeXT-Video](model_doc/llava_next_video)              |       ✅        |         ❌         |      ❌      |
|                    [Longformer](model_doc/longformer)                    |       ✅        |         ✅         |      ❌      |
|                        [LongT5](model_doc/longt5)                        |       ✅        |         ❌         |      ✅      |
|                          [LUKE](model_doc/luke)                          |       ✅        |         ❌         |      ❌      |
|                        [LXMERT](model_doc/lxmert)                        |       ✅        |         ✅         |      ❌      |
|


key: _node_content
string_value:
{"id_": "aaa980f0-e175-45ec-b693-d3abfd96e34f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "832b5863-6a7d-4fe5-889d-679f91578cf0", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "27d412a9e79be4a05c5466b3bcb3b7cdbabf57fdad8382df0bbb97b7611b56cf", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ed9fbda6-f888-47a5-a408-933187b55c9d", "node_type": "1", "metadata": {}, "hash": "384bd0c0909cd622a450384c95fb46eb1787cca08e5caa2df65b300e0139d686", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 20250, "end_char_idx": 21174, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
|       ✅        |         ✅         |      ❌      |
|                        [LongT5](model_doc/longt5)                        |       ✅        |         ❌         |      ✅      |
|                          [LUKE](model_doc/luke)                          |       ✅        |         ❌         |      ❌      |
|                        [LXMERT](model_doc/lxmert)                        |       ✅        |         ✅         |      ❌      |
|                        [M-CTC-T](model_doc/mctct)                        |       ✅        |         ❌         |      ❌      |
|                       [M2M100](model_doc/m2m_100)                        |       ✅        |         ❌         |      ❌      |
|                    [MADLAD-400](model_doc/madlad-400)                    |       ✅        |         ✅         |      ✅      |
|                         [Mamba](model_doc/mamba)                         |       ✅        |         ❌


key: _node_content
string_value:
{"id_": "ed9fbda6-f888-47a5-a408-933187b55c9d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "aaa980f0-e175-45ec-b693-d3abfd96e34f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "0bc0e16efec0fb95437d360a8ddb211ae2f984070afbf2dda326fa5c29da7d87", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4bb3c88a-960f-4880-a4ab-975c127be548", "node_type": "1", "metadata": {}, "hash": "27f5c52ec20a243135233a5bdb78a7375386ec10db408ab3fc4d29009936c072", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 20712, "end_char_idx": 21659, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[M-CTC-T](model_doc/mctct)                        |       ✅        |         ❌         |      ❌      |
|                       [M2M100](model_doc/m2m_100)                        |       ✅        |         ❌         |      ❌      |
|                    [MADLAD-400](model_doc/madlad-400)                    |       ✅        |         ✅         |      ✅      |
|                         [Mamba](model_doc/mamba)                         |       ✅        |         ❌         |      ❌      |
|                        [mamba2](model_doc/mamba2)                        |       ✅        |         ❌         |      ❌      |
|                        [Marian](model_doc/marian)                        |       ✅        |         ✅         |      ✅      |
|                      [MarkupLM](model_doc/markuplm)                      |       ✅        |         ❌         |      ❌      |
|                   [Mask2Former](model_doc/mask2former)                   |


key: _node_content
string_value:
{"id_": "4bb3c88a-960f-4880-a4ab-975c127be548", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ed9fbda6-f888-47a5-a408-933187b55c9d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "5eb7e29245b597384ded6cf54cedecd932be35fa561e356bd5c9c9a2b22b778f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "378b4505-1d37-4bd2-99ec-ed58ccd4374b", "node_type": "1", "metadata": {}, "hash": "e9945f63403b1288c5321c233d5863c37f85c7e9845ffb3ce45c720bbda00eca", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 21163, "end_char_idx": 22096, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
|         ❌         |      ❌      |
|                        [mamba2](model_doc/mamba2)                        |       ✅        |         ❌         |      ❌      |
|                        [Marian](model_doc/marian)                        |       ✅        |         ✅         |      ✅      |
|                      [MarkupLM](model_doc/markuplm)                      |       ✅        |         ❌         |      ❌      |
|                   [Mask2Former](model_doc/mask2former)                   |       ✅        |         ❌         |      ❌      |
|                    [MaskFormer](model_doc/maskformer)                    |       ✅        |         ❌         |      ❌      |
|                        [MatCha](model_doc/matcha)                        |       ✅        |         ❌         |      ❌      |
|                         [mBART](model_doc/mbart)                         |       ✅        |         ✅         |      ✅      |
|


key: _node_content
string_value:
{"id_": "378b4505-1d37-4bd2-99ec-ed58ccd4374b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4bb3c88a-960f-4880-a4ab-975c127be548", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "2f0ed1c045133e58c04c8d98ce7594ee813f6c9562f53d787cbe331c2e21f345", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "03992253-6766-42d4-a3e2-e3bd9c239309", "node_type": "1", "metadata": {}, "hash": "843af70398ed4aec2be4f4c5792a6562eadf3d6d310f980033fd89cf800743a5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 21658, "end_char_idx": 22599, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
|       ✅        |         ❌         |      ❌      |
|                    [MaskFormer](model_doc/maskformer)                    |       ✅        |         ❌         |      ❌      |
|                        [MatCha](model_doc/matcha)                        |       ✅        |         ❌         |      ❌      |
|                         [mBART](model_doc/mbart)                         |       ✅        |         ✅         |      ✅      |
|                      [mBART-50](model_doc/mbart50)                       |       ✅        |         ✅         |      ✅      |
|                          [MEGA](model_doc/mega)                          |       ✅        |         ❌         |      ❌      |
|                 [Megatron-BERT](model_doc/megatron-bert)                 |       ✅        |         ❌         |      ❌      |
|                 [Megatron-GPT2](model_doc/megatron_gpt2)                 |       ✅        |         ✅         |      ✅


key: _node_content
string_value:
{"id_": "03992253-6766-42d4-a3e2-e3bd9c239309", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "378b4505-1d37-4bd2-99ec-ed58ccd4374b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "eda9d073e889a372c3dea806730c3985d535343ec20ffaab43d505339ec17e9c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8a6f266d-0a05-4ef0-8da1-0584f4cc6879", "node_type": "1", "metadata": {}, "hash": "3c059c2a06d3a7305565033ea80137a2dc8611a2e914bdbfe95343edc6cddbe8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 22118, "end_char_idx": 23075, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[mBART-50](model_doc/mbart50)                       |       ✅        |         ✅         |      ✅      |
|                          [MEGA](model_doc/mega)                          |       ✅        |         ❌         |      ❌      |
|                 [Megatron-BERT](model_doc/megatron-bert)                 |       ✅        |         ❌         |      ❌      |
|                 [Megatron-GPT2](model_doc/megatron_gpt2)                 |       ✅        |         ✅         |      ✅      |
|                       [MGP-STR](model_doc/mgp-str)                       |       ✅        |         ❌         |      ❌      |
|                       [Mistral](model_doc/mistral)                       |       ✅        |         ✅         |      ✅      |
|                       [Mixtral](model_doc/mixtral)                       |       ✅        |         ❌         |      ❌      |
|                         [mLUKE](model_doc/mluke)                         |       ✅


key: _node_content
string_value:
{"id_": "8a6f266d-0a05-4ef0-8da1-0584f4cc6879", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "03992253-6766-42d4-a3e2-e3bd9c239309", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "57bab3a5ee49ea97e37c780e1c181e2a0697504a357771adb5581a9b9b5100d3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a8637029-957b-4961-a88b-b1995717f30b", "node_type": "1", "metadata": {}, "hash": "3aec3b261687b1f5380350cc1e9275172c9fe28e8b43087aac6c358554312977", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 22591, "end_char_idx": 23559, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
|      ✅      |
|                       [MGP-STR](model_doc/mgp-str)                       |       ✅        |         ❌         |      ❌      |
|                       [Mistral](model_doc/mistral)                       |       ✅        |         ✅         |      ✅      |
|                       [Mixtral](model_doc/mixtral)                       |       ✅        |         ❌         |      ❌      |
|                         [mLUKE](model_doc/mluke)                         |       ✅        |         ❌         |      ❌      |
|                           [MMS](model_doc/mms)                           |       ✅        |         ✅         |      ✅      |
|                    [MobileBERT](model_doc/mobilebert)                    |       ✅        |         ✅         |      ❌      |
|                  [MobileNetV1](model_doc/mobilenet_v1)                   |       ✅        |         ❌         |      ❌      |
|                  [MobileNetV2](model_doc/mobilenet_v2)


key: _node_content
string_value:
{"id_": "a8637029-957b-4961-a88b-b1995717f30b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8a6f266d-0a05-4ef0-8da1-0584f4cc6879", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "f13d16ca5d3110d9e7db27344eb24788f6689372a0957e4bcb76cf43eb70e719", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d43bea50-7fb6-4e6c-8234-8bcf06a32a63", "node_type": "1", "metadata": {}, "hash": "d33c38c64386fafda00cea812a9b7346e5b8efa0f1dd3c2348c050247fae9f89", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 23066, "end_char_idx": 24016, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
|       ✅        |         ❌         |      ❌      |
|                           [MMS](model_doc/mms)                           |       ✅        |         ✅         |      ✅      |
|                    [MobileBERT](model_doc/mobilebert)                    |       ✅        |         ✅         |      ❌      |
|                  [MobileNetV1](model_doc/mobilenet_v1)                   |       ✅        |         ❌         |      ❌      |
|                  [MobileNetV2](model_doc/mobilenet_v2)                   |       ✅        |         ❌         |      ❌      |
|                     [MobileViT](model_doc/mobilevit)                     |       ✅        |         ✅         |      ❌      |
|                   [MobileViTV2](model_doc/mobilevitv2)                   |       ✅        |         ❌         |      ❌      |
|                         [MPNet](model_doc/mpnet)                         |       ✅        |         ✅         |      ❌      |
|


key: _node_content
string_value:
{"id_": "d43bea50-7fb6-4e6c-8234-8bcf06a32a63", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a8637029-957b-4961-a88b-b1995717f30b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "741de6302bf72a8cd2910db23f13518d33e284ae25271cadbcd5fb272f125e18", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d2d937c4-ab55-4796-bdd9-ba391a86c9af", "node_type": "1", "metadata": {}, "hash": "23e2ab7ef252db4597a8444c71ec56eca712691e20d3fb19b39408e19486df90", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 23578, "end_char_idx": 24502, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
|       ✅        |         ❌         |      ❌      |
|                     [MobileViT](model_doc/mobilevit)                     |       ✅        |         ✅         |      ❌      |
|                   [MobileViTV2](model_doc/mobilevitv2)                   |       ✅        |         ❌         |      ❌      |
|                         [MPNet](model_doc/mpnet)                         |       ✅        |         ✅         |      ❌      |
|                           [MPT](model_doc/mpt)                           |       ✅        |         ❌         |      ❌      |
|                           [MRA](model_doc/mra)                           |       ✅        |         ❌         |      ❌      |
|                           [MT5](model_doc/mt5)                           |       ✅        |         ✅         |      ✅      |
|                      [MusicGen](model_doc/musicgen)                      |       ✅        |         ❌


key: _node_content
string_value:
{"id_": "d2d937c4-ab55-4796-bdd9-ba391a86c9af", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d43bea50-7fb6-4e6c-8234-8bcf06a32a63", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "c8834468aed62e86bc9694607669958b7f6f3bc208213d3d61677f28ad10eef1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f0de7de4-1e13-4e20-9201-b46e23ebfd21", "node_type": "1", "metadata": {}, "hash": "e052ec8007c8e808f4c59a6ca7cadb2e66a589f3858b78e5b4f3c84a7fa48e47", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 24043, "end_char_idx": 24964, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[MPT](model_doc/mpt)                           |       ✅        |         ❌         |      ❌      |
|                           [MRA](model_doc/mra)                           |       ✅        |         ❌         |      ❌      |
|                           [MT5](model_doc/mt5)                           |       ✅        |         ✅         |      ✅      |
|                      [MusicGen](model_doc/musicgen)                      |       ✅        |         ❌         |      ❌      |
|               [MusicGen Melody](model_doc/musicgen_melody)               |       ✅        |         ❌         |      ❌      |
|                           [MVP](model_doc/mvp)                           |       ✅        |         ❌         |      ❌      |
|                           [NAT](model_doc/nat)                           |       ✅        |         ❌         |      ❌      |
|                      [Nemotron](model_doc/nemotron)


key: _node_content
string_value:
{"id_": "f0de7de4-1e13-4e20-9201-b46e23ebfd21", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d2d937c4-ab55-4796-bdd9-ba391a86c9af", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "452e14e0dd9687f3b4ab99ea9404ddc0860fb96dca0a31c64f44e8de5462d122", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "83749417-09db-4eef-8187-2c2ef9706a9b", "node_type": "1", "metadata": {}, "hash": "02df87c270888648a339858e489a587a19b57a4b917b8e25ed4d0e000eb259b3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 24491, "end_char_idx": 25424, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
|         ❌         |      ❌      |
|               [MusicGen Melody](model_doc/musicgen_melody)               |       ✅        |         ❌         |      ❌      |
|                           [MVP](model_doc/mvp)                           |       ✅        |         ❌         |      ❌      |
|                           [NAT](model_doc/nat)                           |       ✅        |         ❌         |      ❌      |
|                      [Nemotron](model_doc/nemotron)                      |       ✅        |         ❌         |      ❌      |
|                         [Nezha](model_doc/nezha)                         |       ✅        |         ❌         |      ❌      |
|                          [NLLB](model_doc/nllb)                          |       ✅        |         ❌         |      ❌      |
|                      [NLLB-MOE](model_doc/nllb-moe)                      |       ✅        |         ❌         |      ❌      |
|


key: _node_content
string_value:
{"id_": "83749417-09db-4eef-8187-2c2ef9706a9b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f0de7de4-1e13-4e20-9201-b46e23ebfd21", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "3cc0398a46ff7e353bd50c32c16ee9b96f98ed430ed695f378668aba444ced17", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "27f0d062-c20a-4e9e-ac9a-e9c1b76cc9a0", "node_type": "1", "metadata": {}, "hash": "ec015772ac15c63f77357545f554e0fe78e11e76952c0032ae91aac1f16cb2ed", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 24986, "end_char_idx": 25910, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
|       ✅        |         ❌         |      ❌      |
|                         [Nezha](model_doc/nezha)                         |       ✅        |         ❌         |      ❌      |
|                          [NLLB](model_doc/nllb)                          |       ✅        |         ❌         |      ❌      |
|                      [NLLB-MOE](model_doc/nllb-moe)                      |       ✅        |         ❌         |      ❌      |
|                        [Nougat](model_doc/nougat)                        |       ✅        |         ✅         |      ✅      |
|                 [Nyströmformer](model_doc/nystromformer)                 |       ✅        |         ❌         |      ❌      |
|                          [OLMo](model_doc/olmo)                          |       ✅        |         ❌         |      ❌      |
|                     [OneFormer](model_doc/oneformer)                     |       ✅        |         ❌


key: _node_content
string_value:
{"id_": "27f0d062-c20a-4e9e-ac9a-e9c1b76cc9a0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "83749417-09db-4eef-8187-2c2ef9706a9b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "69df8871cb928c12d0fde73a76d0b3e0f79cc3e88b4f549cb40221015be179fe", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f40ab089-d376-4392-983a-e8238beaf8bf", "node_type": "1", "metadata": {}, "hash": "1260f2284fbe2cdf23cbcf47961d42ef3b5c4f08c8cee7166705ccd28a51b51b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 25421, "end_char_idx": 26395, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
|
|                        [Nougat](model_doc/nougat)                        |       ✅        |         ✅         |      ✅      |
|                 [Nyströmformer](model_doc/nystromformer)                 |       ✅        |         ❌         |      ❌      |
|                          [OLMo](model_doc/olmo)                          |       ✅        |         ❌         |      ❌      |
|                     [OneFormer](model_doc/oneformer)                     |       ✅        |         ❌         |      ❌      |
|                    [OpenAI GPT](model_doc/openai-gpt)                    |       ✅        |         ✅         |      ❌      |
|                      [OpenAI GPT-2](model_doc/gpt2)                      |       ✅        |         ✅         |      ✅      |
|                    [OpenLlama](model_doc/open-llama)                     |       ✅        |         ❌         |      ❌      |
|                           [OPT](model_doc/opt)                           |


key: _node_content
string_value:
{"id_": "f40ab089-d376-4392-983a-e8238beaf8bf", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "27f0d062-c20a-4e9e-ac9a-e9c1b76cc9a0", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "6d4dc0158119fc11effa53c96cb8de80ca94f368168f311272a63753ef330073", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fa841613-98c6-4a13-a387-917f14d3b360", "node_type": "1", "metadata": {}, "hash": "fddd28d85bddfd5ab0be354f3136b5a3e4e1d8e14ecc8f8361ac2d64c4c57510", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 25909, "end_char_idx": 26832, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
❌         |      ❌      |
|                    [OpenAI GPT](model_doc/openai-gpt)                    |       ✅        |         ✅         |      ❌      |
|                      [OpenAI GPT-2](model_doc/gpt2)                      |       ✅        |         ✅         |      ✅      |
|                    [OpenLlama](model_doc/open-llama)                     |       ✅        |         ❌         |      ❌      |
|                           [OPT](model_doc/opt)                           |       ✅        |         ✅         |      ✅      |
|                       [OWL-ViT](model_doc/owlvit)                        |       ✅        |         ❌         |      ❌      |
|                         [OWLv2](model_doc/owlv2)                         |       ✅        |         ❌         |      ❌      |
|                     [PaliGemma](model_doc/paligemma)                     |       ✅        |         ❌         |      ❌      |
|


key: _node_content
string_value:
{"id_": "fa841613-98c6-4a13-a387-917f14d3b360", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f40ab089-d376-4392-983a-e8238beaf8bf", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "0abc5e657d480b039e521ce53e3a4252768f6c3f238a331b6d909b42afa863ec", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "75356b4e-bd6e-4f6e-9636-09e28e85ecd4", "node_type": "1", "metadata": {}, "hash": "c694232f5e44271acb49b206c8cf594c098dbc6e52375a66dcc6faa8c02b3874", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 26394, "end_char_idx": 27335, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
|       ✅        |         ✅         |      ✅      |
|                       [OWL-ViT](model_doc/owlvit)                        |       ✅        |         ❌         |      ❌      |
|                         [OWLv2](model_doc/owlv2)                         |       ✅        |         ❌         |      ❌      |
|                     [PaliGemma](model_doc/paligemma)                     |       ✅        |         ❌         |      ❌      |
|                  [PatchTSMixer](model_doc/patchtsmixer)                  |       ✅        |         ❌         |      ❌      |
|                      [PatchTST](model_doc/patchtst)                      |       ✅        |         ❌         |      ❌      |
|                       [Pegasus](model_doc/pegasus)                       |       ✅        |         ✅         |      ✅      |
|                     [PEGASUS-X](model_doc/pegasus_x)                     |       ✅        |         ❌         |      ❌


key: _node_content
string_value:
{"id_": "75356b4e-bd6e-4f6e-9636-09e28e85ecd4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fa841613-98c6-4a13-a387-917f14d3b360", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "942ff439bf900ceb0647bc9241c0479de71354dbd24c18d66f856dfe985649a0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ba395bb1-d4e8-469a-b6e3-8abd29d4dbc5", "node_type": "1", "metadata": {}, "hash": "b3cd6d923678bddbded590d6c23d80b7b5fa3f2ea4b0141fac25c3b911cda5cb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 26850, "end_char_idx": 27811, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[PatchTSMixer](model_doc/patchtsmixer)                  |       ✅        |         ❌         |      ❌      |
|                      [PatchTST](model_doc/patchtst)                      |       ✅        |         ❌         |      ❌      |
|                       [Pegasus](model_doc/pegasus)                       |       ✅        |         ✅         |      ✅      |
|                     [PEGASUS-X](model_doc/pegasus_x)                     |       ✅        |         ❌         |      ❌      |
|                     [Perceiver](model_doc/perceiver)                     |       ✅        |         ❌         |      ❌      |
|                     [Persimmon](model_doc/persimmon)                     |       ✅        |         ❌         |      ❌      |
|                           [Phi](model_doc/phi)                           |       ✅        |         ❌         |      ❌      |
|                          [Phi3](model_doc/phi3)                          |       ✅


key: _node_content
string_value:
{"id_": "ba395bb1-d4e8-469a-b6e3-8abd29d4dbc5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "75356b4e-bd6e-4f6e-9636-09e28e85ecd4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "1d719ad1c44059e6d508613e9fb62476ab2ac3cd122c929fce5b205faae8d1f7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4482727b-6a2b-4b7b-830e-617747a92e03", "node_type": "1", "metadata": {}, "hash": "bf780a405d0fa7b36e52c0e5912981d2c0c21d1f00c46b0bfb2cb2adce0bf7ae", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 27327, "end_char_idx": 28294, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
|      ❌      |
|                     [Perceiver](model_doc/perceiver)                     |       ✅        |         ❌         |      ❌      |
|                     [Persimmon](model_doc/persimmon)                     |       ✅        |         ❌         |      ❌      |
|                           [Phi](model_doc/phi)                           |       ✅        |         ❌         |      ❌      |
|                          [Phi3](model_doc/phi3)                          |       ✅        |         ❌         |      ❌      |
|                       [PhoBERT](model_doc/phobert)                       |       ✅        |         ✅         |      ✅      |
|                    [Pix2Struct](model_doc/pix2struct)                    |       ✅        |         ❌         |      ❌      |
|                        [PLBart](model_doc/plbart)                        |       ✅        |         ❌         |      ❌      |
|                    [PoolFormer](model_doc/poolformer)


key: _node_content
string_value:
{"id_": "4482727b-6a2b-4b7b-830e-617747a92e03", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ba395bb1-d4e8-469a-b6e3-8abd29d4dbc5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "dcc46b56f4f692822ccefda5b15e0669d458fb28f868978ae446717dc4c5ff92", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "23abf4d5-a24a-4739-bb71-2825cb9de674", "node_type": "1", "metadata": {}, "hash": "384dbe82566148709e7420b888de79edf66afdb239f9a50b38f95f1e99e053a0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 27802, "end_char_idx": 28752, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
|       ✅        |         ❌         |      ❌      |
|                       [PhoBERT](model_doc/phobert)                       |       ✅        |         ✅         |      ✅      |
|                    [Pix2Struct](model_doc/pix2struct)                    |       ✅        |         ❌         |      ❌      |
|                        [PLBart](model_doc/plbart)                        |       ✅        |         ❌         |      ❌      |
|                    [PoolFormer](model_doc/poolformer)                    |       ✅        |         ❌         |      ❌      |
|                     [Pop2Piano](model_doc/pop2piano)                     |       ✅        |         ❌         |      ❌      |
|                    [ProphetNet](model_doc/prophetnet)                    |       ✅        |         ❌         |      ❌      |
|                           [PVT](model_doc/pvt)                           |       ✅        |         ❌         |      ❌      |
|


key: _node_content
string_value:
{"id_": "23abf4d5-a24a-4739-bb71-2825cb9de674", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4482727b-6a2b-4b7b-830e-617747a92e03", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "d72769146ea90edbf4994cdcabbb71af2887007e8f5b44bb8c599db433aa0db3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4ef08cf1-d620-47c6-82ae-51ddb19e87cd", "node_type": "1", "metadata": {}, "hash": "9ed5082c7ae74973b950749e33a0b60de23f0faf523e6521421ce7cb449ccd55", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 28314, "end_char_idx": 29238, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
|       ✅        |         ❌         |      ❌      |
|                     [Pop2Piano](model_doc/pop2piano)                     |       ✅        |         ❌         |      ❌      |
|                    [ProphetNet](model_doc/prophetnet)                    |       ✅        |         ❌         |      ❌      |
|                           [PVT](model_doc/pvt)                           |       ✅        |         ❌         |      ❌      |
|                        [PVTv2](model_doc/pvt_v2)                         |       ✅        |         ❌         |      ❌      |
|                       [QDQBert](model_doc/qdqbert)                       |       ✅        |         ❌         |      ❌      |
|                         [Qwen2](model_doc/qwen2)                         |       ✅        |         ❌         |      ❌      |
|                   [Qwen2Audio](model_doc/qwen2_audio)                    |       ✅        |         ❌


key: _node_content
string_value:
{"id_": "4ef08cf1-d620-47c6-82ae-51ddb19e87cd", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "23abf4d5-a24a-4739-bb71-2825cb9de674", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "f97ee59556df34b4a51095d3a4ab7cea69d291e0aaa14ae9301dfbbd5f1e3bd7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d7d63cfd-cf88-4b0a-a2d6-58c6aa192ab1", "node_type": "1", "metadata": {}, "hash": "e92019d9c8b9c6c78476b29af5897d6c47a0631747ad8eeccfc7e70b0328d5cf", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 28776, "end_char_idx": 29697, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[PVTv2](model_doc/pvt_v2)                         |       ✅        |         ❌         |      ❌      |
|                       [QDQBert](model_doc/qdqbert)                       |       ✅        |         ❌         |      ❌      |
|                         [Qwen2](model_doc/qwen2)                         |       ✅        |         ❌         |      ❌      |
|                   [Qwen2Audio](model_doc/qwen2_audio)                    |       ✅        |         ❌         |      ❌      |
|                     [Qwen2MoE](model_doc/qwen2_moe)                      |       ✅        |         ❌         |      ❌      |
|                      [Qwen2VL](model_doc/qwen2_vl)                       |       ✅        |         ❌         |      ❌      |
|                           [RAG](model_doc/rag)                           |       ✅        |         ✅         |      ❌      |
|                         [REALM](model_doc/realm)


key: _node_content
string_value:
{"id_": "d7d63cfd-cf88-4b0a-a2d6-58c6aa192ab1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4ef08cf1-d620-47c6-82ae-51ddb19e87cd", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "f3ecbc20595e4bd3c634cd5ea5472d871d48b2725ff1c56610d8c4e8084deb2f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e0c2e932-dfc9-40bf-a42f-580d33ae70ec", "node_type": "1", "metadata": {}, "hash": "32ccbcef1609fc9efcb9c990ebec3d80525ae4dc3c3f5f26b19489d168c69657", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 29237, "end_char_idx": 30160, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
❌         |      ❌      |
|                     [Qwen2MoE](model_doc/qwen2_moe)                      |       ✅        |         ❌         |      ❌      |
|                      [Qwen2VL](model_doc/qwen2_vl)                       |       ✅        |         ❌         |      ❌      |
|                           [RAG](model_doc/rag)                           |       ✅        |         ✅         |      ❌      |
|                         [REALM](model_doc/realm)                         |       ✅        |         ❌         |      ❌      |
|               [RecurrentGemma](model_doc/recurrent_gemma)                |       ✅        |         ❌         |      ❌      |
|                      [Reformer](model_doc/reformer)                      |       ✅        |         ❌         |      ❌      |
|                        [RegNet](model_doc/regnet)                        |       ✅        |         ✅         |      ✅      |
|


key: _node_content
string_value:
{"id_": "e0c2e932-dfc9-40bf-a42f-580d33ae70ec", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d7d63cfd-cf88-4b0a-a2d6-58c6aa192ab1", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "afd1627485d84e9917f4f5ddecc10d24585796ed94fc951cc55e3124609fa3a2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "43e15f9b-f030-4998-80d1-0edcdf726449", "node_type": "1", "metadata": {}, "hash": "f2c0b6c9b04469ba1f152045063cb4e2233f528c8c3d3aa1c30e0ea08478245c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 29722, "end_char_idx": 30672, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
|       ✅        |         ❌         |      ❌      |
|               [RecurrentGemma](model_doc/recurrent_gemma)                |       ✅        |         ❌         |      ❌      |
|                      [Reformer](model_doc/reformer)                      |       ✅        |         ❌         |      ❌      |
|                        [RegNet](model_doc/regnet)                        |       ✅        |         ✅         |      ✅      |
|                       [RemBERT](model_doc/rembert)                       |       ✅        |         ✅         |      ❌      |
|                        [ResNet](model_doc/resnet)                        |       ✅        |         ✅         |      ✅      |
|                     [RetriBERT](model_doc/retribert)                     |       ✅        |         ❌         |      ❌      |
|                       [RoBERTa](model_doc/roberta)                       |       ✅        |         ✅         |      ✅      |
|


key: _node_content
string_value:
{"id_": "43e15f9b-f030-4998-80d1-0edcdf726449", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e0c2e932-dfc9-40bf-a42f-580d33ae70ec", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "6a49f1d6a5dc2871665fdfaca08c9b32b02d5c01047a3abcdfd5328bc5628b5e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4dd29fdb-338c-4517-90ca-f74565e8638a", "node_type": "1", "metadata": {}, "hash": "019b699020d178e44e04594b5f9ab0e626efb2e792bcf2e7e4e420687f99bf25", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 30183, "end_char_idx": 31158, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[RemBERT](model_doc/rembert)                       |       ✅        |         ✅         |      ❌      |
|                        [ResNet](model_doc/resnet)                        |       ✅        |         ✅         |      ✅      |
|                     [RetriBERT](model_doc/retribert)                     |       ✅        |         ❌         |      ❌      |
|                       [RoBERTa](model_doc/roberta)                       |       ✅        |         ✅         |      ✅      |
|          [RoBERTa-PreLayerNorm](model_doc/roberta-prelayernorm)          |       ✅        |         ✅         |      ✅      |
|                      [RoCBert](model_doc/roc_bert)                       |       ✅        |         ❌         |      ❌      |
|                      [RoFormer](model_doc/roformer)                      |       ✅        |         ✅         |      ✅      |
|                       [RT-DETR](model_doc/rt_detr)                       |       ✅        |         ❌


key: _node_content
string_value:
{"id_": "4dd29fdb-338c-4517-90ca-f74565e8638a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "43e15f9b-f030-4998-80d1-0edcdf726449", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "c130f9c3f757b20be9e64852355ad1f294fa2cdf8f9458aa08a6821d173bd185", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "78873839-4610-4387-9d81-c80f3ca5c93e", "node_type": "1", "metadata": {}, "hash": "4ec5b0434c6faf00513bea372339509a03fcb25901b4105b5ee4f2ecfb66fac9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 30662, "end_char_idx": 31643, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
✅      |
|          [RoBERTa-PreLayerNorm](model_doc/roberta-prelayernorm)          |       ✅        |         ✅         |      ✅      |
|                      [RoCBert](model_doc/roc_bert)                       |       ✅        |         ❌         |      ❌      |
|                      [RoFormer](model_doc/roformer)                      |       ✅        |         ✅         |      ✅      |
|                       [RT-DETR](model_doc/rt_detr)                       |       ✅        |         ❌         |      ❌      |
|                [RT-DETR-ResNet](model_doc/rt_detr_resnet)                |       ✅        |         ❌         |      ❌      |
|                          [RWKV](model_doc/rwkv)                          |       ✅        |         ❌         |      ❌      |
|                           [SAM](model_doc/sam)                           |       ✅        |         ✅         |      ❌      |
|                  [SeamlessM4T](model_doc/seamless_m4t)                   |


key: _node_content
string_value:
{"id_": "78873839-4610-4387-9d81-c80f3ca5c93e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4dd29fdb-338c-4517-90ca-f74565e8638a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "5c4c245f021b3e0485c5e5acd2d934aa4fb4d07aa6876d43c41bd821014a4ffb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1aa287e6-4b5b-4981-8abb-4f67ed7da7ba", "node_type": "1", "metadata": {}, "hash": "c5ce05de614ff35cfe5a3324ef771c85af18917bfec2ca8357fbd25e77813998", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 31157, "end_char_idx": 32080, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
❌         |      ❌      |
|                [RT-DETR-ResNet](model_doc/rt_detr_resnet)                |       ✅        |         ❌         |      ❌      |
|                          [RWKV](model_doc/rwkv)                          |       ✅        |         ❌         |      ❌      |
|                           [SAM](model_doc/sam)                           |       ✅        |         ✅         |      ❌      |
|                  [SeamlessM4T](model_doc/seamless_m4t)                   |       ✅        |         ❌         |      ❌      |
|                [SeamlessM4Tv2](model_doc/seamless_m4t_v2)                |       ✅        |         ❌         |      ❌      |
|                     [SegFormer](model_doc/segformer)                     |       ✅        |         ✅         |      ❌      |
|                        [SegGPT](model_doc/seggpt)                        |       ✅        |         ❌         |      ❌      |
|


key: _node_content
string_value:
{"id_": "1aa287e6-4b5b-4981-8abb-4f67ed7da7ba", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "78873839-4610-4387-9d81-c80f3ca5c93e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "a93120b94f625dca634c08b76772d8c1ade560604aa0b3bbbeffd69a326602b5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0635ae74-9e2c-45f1-a46c-a12b2a5493dd", "node_type": "1", "metadata": {}, "hash": "2e7df9f617f7f71ec329cf50538bb908b7b39d3eff24e5e88a32f3fb0b22ea03", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 31642, "end_char_idx": 32592, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
|       ✅        |         ❌         |      ❌      |
|                [SeamlessM4Tv2](model_doc/seamless_m4t_v2)                |       ✅        |         ❌         |      ❌      |
|                     [SegFormer](model_doc/segformer)                     |       ✅        |         ✅         |      ❌      |
|                        [SegGPT](model_doc/seggpt)                        |       ✅        |         ❌         |      ❌      |
|                           [SEW](model_doc/sew)                           |       ✅        |         ❌         |      ❌      |
|                         [SEW-D](model_doc/sew-d)                         |       ✅        |         ❌         |      ❌      |
|                        [SigLIP](model_doc/siglip)                        |       ✅        |         ❌         |      ❌      |
|        [Speech Encoder decoder](model_doc/speech-encoder-decoder)        |       ✅        |         ❌         |      ✅      |
|


key: _node_content
string_value:
{"id_": "0635ae74-9e2c-45f1-a46c-a12b2a5493dd", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1aa287e6-4b5b-4981-8abb-4f67ed7da7ba", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "d7e8d401411696c5c688d1f86d20ba90c43eee54bbf44324642b4d22d9180110", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9396e42f-260c-4a3c-9889-48afd916f096", "node_type": "1", "metadata": {}, "hash": "822a4564d1704ae55a4c997e200e4150062818052105cbaf50191893fde9d75c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 32107, "end_char_idx": 33095, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[SEW](model_doc/sew)                           |       ✅        |         ❌         |      ❌      |
|                         [SEW-D](model_doc/sew-d)                         |       ✅        |         ❌         |      ❌      |
|                        [SigLIP](model_doc/siglip)                        |       ✅        |         ❌         |      ❌      |
|        [Speech Encoder decoder](model_doc/speech-encoder-decoder)        |       ✅        |         ❌         |      ✅      |
|                 [Speech2Text](model_doc/speech_to_text)                  |       ✅        |         ✅         |      ❌      |
|                      [SpeechT5](model_doc/speecht5)                      |       ✅        |         ❌         |      ❌      |
|                      [Splinter](model_doc/splinter)                      |       ✅        |         ❌         |      ❌      |
|                   [SqueezeBERT](model_doc/squeezebert)                   |       ✅        |         ❌         |      ❌


key: _node_content
string_value:
{"id_": "9396e42f-260c-4a3c-9889-48afd916f096", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0635ae74-9e2c-45f1-a46c-a12b2a5493dd", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "8515edfb9599bbfa80022219403de86aa1caa4c52bbcc1c9885f146448512209", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f7ef4897-53ab-4e7e-a19b-140a59a23920", "node_type": "1", "metadata": {}, "hash": "c4bb5caa24e9c043bb19d2deac3b99031092f60c339ca7b8d1d9cc104d79d86c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 32589, "end_char_idx": 33590, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
|
|                 [Speech2Text](model_doc/speech_to_text)                  |       ✅        |         ✅         |      ❌      |
|                      [SpeechT5](model_doc/speecht5)                      |       ✅        |         ❌         |      ❌      |
|                      [Splinter](model_doc/splinter)                      |       ✅        |         ❌         |      ❌      |
|                   [SqueezeBERT](model_doc/squeezebert)                   |       ✅        |         ❌         |      ❌      |
|                      [StableLm](model_doc/stablelm)                      |       ✅        |         ❌         |      ❌      |
|                    [Starcoder2](model_doc/starcoder2)                    |       ✅        |         ❌         |      ❌      |
|                    [SuperPoint](model_doc/superpoint)                    |       ✅        |         ❌         |      ❌      |
|                   [SwiftFormer](model_doc/swiftformer)                   |       ✅        |         ✅


key: _node_content
string_value:
{"id_": "f7ef4897-53ab-4e7e-a19b-140a59a23920", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9396e42f-260c-4a3c-9889-48afd916f096", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "9c909a3163175172d5643009f58fa50d6674c54647aaf4961f54392b794b555b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9be8f067-cbb6-4887-b371-fe258d10aff9", "node_type": "1", "metadata": {}, "hash": "f2877feeaec504a6c6e2a3c73f3417f8a6473b361391298be4c45026ab7b86c3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 33087, "end_char_idx": 34112, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
|      ❌      |
|                      [StableLm](model_doc/stablelm)                      |       ✅        |         ❌         |      ❌      |
|                    [Starcoder2](model_doc/starcoder2)                    |       ✅        |         ❌         |      ❌      |
|                    [SuperPoint](model_doc/superpoint)                    |       ✅        |         ❌         |      ❌      |
|                   [SwiftFormer](model_doc/swiftformer)                   |       ✅        |         ✅         |      ❌      |
|                    [Swin Transformer](model_doc/swin)                    |       ✅        |         ✅         |      ❌      |
|                 [Swin Transformer V2](model_doc/swinv2)                  |       ✅        |         ❌         |      ❌      |
|                       [Swin2SR](model_doc/swin2sr)                       |       ✅        |         ❌         |      ❌      |
|           [SwitchTransformers](model_doc/switch_transformers)            |       ✅        |         ❌         |


key: _node_content
string_value:
{"id_": "9be8f067-cbb6-4887-b371-fe258d10aff9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f7ef4897-53ab-4e7e-a19b-140a59a23920", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "7174e03975b4e13bad6c53d72b7ae8fa1ac76cbcb0c571f6a2b2fb06d6ba432f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "db804067-5407-4f3a-a168-e4674fbb1f10", "node_type": "1", "metadata": {}, "hash": "b9ab9dd487a49cff562d2fff47d2e1b33c8e423b65832f8615d5fa50907b0c68", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 33589, "end_char_idx": 34587, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
✅         |      ❌      |
|                    [Swin Transformer](model_doc/swin)                    |       ✅        |         ✅         |      ❌      |
|                 [Swin Transformer V2](model_doc/swinv2)                  |       ✅        |         ❌         |      ❌      |
|                       [Swin2SR](model_doc/swin2sr)                       |       ✅        |         ❌         |      ❌      |
|           [SwitchTransformers](model_doc/switch_transformers)            |       ✅        |         ❌         |      ❌      |
|                            [T5](model_doc/t5)                            |       ✅        |         ✅         |      ✅      |
|                        [T5v1.1](model_doc/t5v1.1)                        |       ✅        |         ✅         |      ✅      |
|             [Table Transformer](model_doc/table-transformer)             |       ✅        |         ❌         |      ❌      |
|                         [TAPAS](model_doc/tapas)                         |


key: _node_content
string_value:
{"id_": "db804067-5407-4f3a-a168-e4674fbb1f10", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9be8f067-cbb6-4887-b371-fe258d10aff9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "8e02c63ffe70e1f392f6e8a9e3e25366aed389487369e38439e8bd3edf85fa99", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "616e120c-3ae0-4995-b64b-1c854430bde4", "node_type": "1", "metadata": {}, "hash": "027b69a25bf24c89add0ba2e7d03163f89b7e81809d7ea1de6dfcb061c85b67f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 34101, "end_char_idx": 35136, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
❌         |      ❌      |
|                            [T5](model_doc/t5)                            |       ✅        |         ✅         |      ✅      |
|                        [T5v1.1](model_doc/t5v1.1)                        |       ✅        |         ✅         |      ✅      |
|             [Table Transformer](model_doc/table-transformer)             |       ✅        |         ❌         |      ❌      |
|                         [TAPAS](model_doc/tapas)                         |       ✅        |         ✅         |      ❌      |
|                         [TAPEX](model_doc/tapex)                         |       ✅        |         ✅         |      ✅      |
|       [Time Series Transformer](model_doc/time_series_transformer)       |       ✅        |         ❌         |      ❌      |
|                   [TimeSformer](model_doc/timesformer)                   |       ✅        |         ❌         |      ❌      |
|        [Trajectory Transformer](model_doc/trajectory_transformer)        |       ✅        |         ❌         |


key: _node_content
string_value:
{"id_": "616e120c-3ae0-4995-b64b-1c854430bde4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "db804067-5407-4f3a-a168-e4674fbb1f10", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "6d9323a182cf3728d3b89c088825abfda6057092a6ebf198c1ef7399e7be4b4f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "77fbcb21-82ee-494e-8d4e-6fa50fe6a5df", "node_type": "1", "metadata": {}, "hash": "14013ebc478453b2a4c34ffc024211f146f3c69405b22f4a77b970d7395bcf59", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 34586, "end_char_idx": 35611, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
|       ✅        |         ✅         |      ❌      |
|                         [TAPEX](model_doc/tapex)                         |       ✅        |         ✅         |      ✅      |
|       [Time Series Transformer](model_doc/time_series_transformer)       |       ✅        |         ❌         |      ❌      |
|                   [TimeSformer](model_doc/timesformer)                   |       ✅        |         ❌         |      ❌      |
|        [Trajectory Transformer](model_doc/trajectory_transformer)        |       ✅        |         ❌         |      ❌      |
|                  [Transformer-XL](model_doc/transfo-xl)                  |       ✅        |         ✅         |      ❌      |
|                         [TrOCR](model_doc/trocr)                         |       ✅        |         ❌         |      ❌      |
|                          [TVLT](model_doc/tvlt)                          |       ✅        |         ❌         |      ❌      |
|                           [TVP](model_doc/tvp)                           |


key: _node_content
string_value:
{"id_": "77fbcb21-82ee-494e-8d4e-6fa50fe6a5df", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "616e120c-3ae0-4995-b64b-1c854430bde4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "0c8e7078deb7981f0503fae079b65d1be283a6abf796016d91e4fe5ff837db2b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c84ea8e7-d01d-4bda-b39b-6a15b3cb04de", "node_type": "1", "metadata": {}, "hash": "491b4dc00cad5496da8d2c82b0716bc45acef676fd5af25ecf1de2462000f434", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 35125, "end_char_idx": 36048, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
❌         |      ❌      |
|                  [Transformer-XL](model_doc/transfo-xl)                  |       ✅        |         ✅         |      ❌      |
|                         [TrOCR](model_doc/trocr)                         |       ✅        |         ❌         |      ❌      |
|                          [TVLT](model_doc/tvlt)                          |       ✅        |         ❌         |      ❌      |
|                           [TVP](model_doc/tvp)                           |       ✅        |         ❌         |      ❌      |
|                          [UDOP](model_doc/udop)                          |       ✅        |         ❌         |      ❌      |
|                           [UL2](model_doc/ul2)                           |       ✅        |         ✅         |      ✅      |
|                          [UMT5](model_doc/umt5)                          |       ✅        |         ❌         |      ❌      |
|


key: _node_content
string_value:
{"id_": "c84ea8e7-d01d-4bda-b39b-6a15b3cb04de", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "77fbcb21-82ee-494e-8d4e-6fa50fe6a5df", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "88c3ce94006d69056399b619f0123c66d31fcd30725322a5ca70a72e157fb54e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b98236be-70e0-43ef-a5bc-560112655701", "node_type": "1", "metadata": {}, "hash": "d1658002720beb10432a8f938a824c7a3cd5aa3f457d74d52e883ed0c7697c60", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 35610, "end_char_idx": 36544, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
|       ✅        |         ❌         |      ❌      |
|                          [UDOP](model_doc/udop)                          |       ✅        |         ❌         |      ❌      |
|                           [UL2](model_doc/ul2)                           |       ✅        |         ✅         |      ✅      |
|                          [UMT5](model_doc/umt5)                          |       ✅        |         ❌         |      ❌      |
|                     [UniSpeech](model_doc/unispeech)                     |       ✅        |         ❌         |      ❌      |
|                 [UniSpeechSat](model_doc/unispeech-sat)                  |       ✅        |         ❌         |      ❌      |
|                       [UnivNet](model_doc/univnet)                       |       ✅        |         ❌         |      ❌      |
|                       [UPerNet](model_doc/upernet)                       |       ✅        |         ❌         |


key: _node_content
string_value:
{"id_": "b98236be-70e0-43ef-a5bc-560112655701", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c84ea8e7-d01d-4bda-b39b-6a15b3cb04de", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "bcc79778acf6b10f8feebfdaf5b514912a1ef70ba0e2b2075df2e388e73a9399", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c69e5afa-4043-48b6-8e4a-49bb635de92d", "node_type": "1", "metadata": {}, "hash": "f6fa1decd338bdcc407c6998a24256d299eb1432e6bf2f9fd923a22d5458881d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 36038, "end_char_idx": 36992, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
❌      |
|                     [UniSpeech](model_doc/unispeech)                     |       ✅        |         ❌         |      ❌      |
|                 [UniSpeechSat](model_doc/unispeech-sat)                  |       ✅        |         ❌         |      ❌      |
|                       [UnivNet](model_doc/univnet)                       |       ✅        |         ❌         |      ❌      |
|                       [UPerNet](model_doc/upernet)                       |       ✅        |         ❌         |      ❌      |
|                           [VAN](model_doc/van)                           |       ✅        |         ❌         |      ❌      |
|                   [VideoLlava](model_doc/video_llava)                    |       ✅        |         ❌         |      ❌      |
|                      [VideoMAE](model_doc/videomae)                      |       ✅        |         ❌         |      ❌      |
|                          [ViLT](model_doc/vilt)


key: _node_content
string_value:
{"id_": "c69e5afa-4043-48b6-8e4a-49bb635de92d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b98236be-70e0-43ef-a5bc-560112655701", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "bdd5d07a7068cbe9b21f65678771ca3b417a193c94921491575e66251ae4423e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d167a09f-5aab-4ea6-a7bd-ec731663f737", "node_type": "1", "metadata": {}, "hash": "31e14a65f1aba868307a74888f31ca8772a0c191032c769a11eae265c921d5d2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 36533, "end_char_idx": 37548, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
❌         |      ❌      |
|                           [VAN](model_doc/van)                           |       ✅        |         ❌         |      ❌      |
|                   [VideoLlava](model_doc/video_llava)                    |       ✅        |         ❌         |      ❌      |
|                      [VideoMAE](model_doc/videomae)                      |       ✅        |         ❌         |      ❌      |
|                          [ViLT](model_doc/vilt)                          |       ✅        |         ❌         |      ❌      |
|                      [VipLlava](model_doc/vipllava)                      |       ✅        |         ❌         |      ❌      |
|        [Vision Encoder decoder](model_doc/vision-encoder-decoder)        |       ✅        |         ✅         |      ✅      |
|       [VisionTextDualEncoder](model_doc/vision-text-dual-encoder)        |       ✅        |         ✅         |      ✅      |
|                   [VisualBERT](model_doc/visual_bert)                    |       ✅        |


key: _node_content
string_value:
{"id_": "d167a09f-5aab-4ea6-a7bd-ec731663f737", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c69e5afa-4043-48b6-8e4a-49bb635de92d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "40d8b1482ce06a8d7dc87540d79da9e9248bf9da732f6bed1e65f6e0469ecbc0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ba17de40-3512-44b3-be2e-13c4a7fd30c1", "node_type": "1", "metadata": {}, "hash": "d29f2d8b062b48dd302acf953682fa5b0347fd55934d18c92208983f513bcd7d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 37018, "end_char_idx": 38018, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
|       ✅        |         ❌         |      ❌      |
|                      [VipLlava](model_doc/vipllava)                      |       ✅        |         ❌         |      ❌      |
|        [Vision Encoder decoder](model_doc/vision-encoder-decoder)        |       ✅        |         ✅         |      ✅      |
|       [VisionTextDualEncoder](model_doc/vision-text-dual-encoder)        |       ✅        |         ✅         |      ✅      |
|                   [VisualBERT](model_doc/visual_bert)                    |       ✅        |         ❌         |      ❌      |
|                           [ViT](model_doc/vit)                           |       ✅        |         ✅         |      ✅      |
|                    [ViT Hybrid](model_doc/vit_hybrid)                    |       ✅        |         ❌         |      ❌      |
|                        [VitDet](model_doc/vitdet)                        |       ✅        |         ❌         |      ❌      |
|                       [ViTMAE](model_doc/vit_mae)


key: _node_content
string_value:
{"id_": "ba17de40-3512-44b3-be2e-13c4a7fd30c1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d167a09f-5aab-4ea6-a7bd-ec731663f737", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "3922004f56c68a92bdecf4141a54f19ba345533fdcb5f026a13b73d899b4bc22", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9e3ca454-92cf-4e3d-8f76-20cfd9f4a11f", "node_type": "1", "metadata": {}, "hash": "ac1c6faa57a9d3a1719c3717895ff5601ce700392c86bfeba740104c43f44cee", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 37547, "end_char_idx": 38480, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
|         ❌         |      ❌      |
|                           [ViT](model_doc/vit)                           |       ✅        |         ✅         |      ✅      |
|                    [ViT Hybrid](model_doc/vit_hybrid)                    |       ✅        |         ❌         |      ❌      |
|                        [VitDet](model_doc/vitdet)                        |       ✅        |         ❌         |      ❌      |
|                       [ViTMAE](model_doc/vit_mae)                        |       ✅        |         ✅         |      ❌      |
|                      [ViTMatte](model_doc/vitmatte)                      |       ✅        |         ❌         |      ❌      |
|                       [ViTMSN](model_doc/vit_msn)                        |       ✅        |         ❌         |      ❌      |
|                          [VITS](model_doc/vits)                          |       ✅        |         ❌         |      ❌      |
|


key: _node_content
string_value:
{"id_": "9e3ca454-92cf-4e3d-8f76-20cfd9f4a11f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ba17de40-3512-44b3-be2e-13c4a7fd30c1", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "d8c6264c5d40cb5dbf27e12e10a061f59ccafae023df33895dc85658e4301dee", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5505ee32-e187-4768-ab83-c4437f5cdba1", "node_type": "1", "metadata": {}, "hash": "151b86a7753c69941c264bb0031df54cbfaed5a9cd06b415ae829824051ca26b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 38042, "end_char_idx": 38966, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
|       ✅        |         ✅         |      ❌      |
|                      [ViTMatte](model_doc/vitmatte)                      |       ✅        |         ❌         |      ❌      |
|                       [ViTMSN](model_doc/vit_msn)                        |       ✅        |         ❌         |      ❌      |
|                          [VITS](model_doc/vits)                          |       ✅        |         ❌         |      ❌      |
|                         [ViViT](model_doc/vivit)                         |       ✅        |         ❌         |      ❌      |
|                      [Wav2Vec2](model_doc/wav2vec2)                      |       ✅        |         ✅         |      ✅      |
|                 [Wav2Vec2-BERT](model_doc/wav2vec2-bert)                 |       ✅        |         ❌         |      ❌      |
|            [Wav2Vec2-Conformer](model_doc/wav2vec2-conformer)            |       ✅        |         ❌


key: _node_content
string_value:
{"id_": "5505ee32-e187-4768-ab83-c4437f5cdba1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9e3ca454-92cf-4e3d-8f76-20cfd9f4a11f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "8dc83ed27cde68be320fb212fa2ab92471eaea71708e2aa0e03816d2249a2ad6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e206fb57-0267-45ba-9044-64ddb0490fa3", "node_type": "1", "metadata": {}, "hash": "c8e09ae160cbf12eea06b75fcdb45246c577d282e58b78e007c2f594848969a0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 38477, "end_char_idx": 39451, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
|
|                         [ViViT](model_doc/vivit)                         |       ✅        |         ❌         |      ❌      |
|                      [Wav2Vec2](model_doc/wav2vec2)                      |       ✅        |         ✅         |      ✅      |
|                 [Wav2Vec2-BERT](model_doc/wav2vec2-bert)                 |       ✅        |         ❌         |      ❌      |
|            [Wav2Vec2-Conformer](model_doc/wav2vec2-conformer)            |       ✅        |         ❌         |      ❌      |
|              [Wav2Vec2Phoneme](model_doc/wav2vec2_phoneme)               |       ✅        |         ✅         |      ✅      |
|                         [WavLM](model_doc/wavlm)                         |       ✅        |         ❌         |      ❌      |
|                       [Whisper](model_doc/whisper)                       |       ✅        |         ✅         |      ✅      |
|                        [X-CLIP](model_doc/xclip)                         |


key: _node_content
string_value:
{"id_": "e206fb57-0267-45ba-9044-64ddb0490fa3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5505ee32-e187-4768-ab83-c4437f5cdba1", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "96805c71c11277cdc9144203dbbd00cafca6055fb625e549ab3f1dd9ba814226", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c2b5b845-88ed-4aea-8162-1c1c5545c969", "node_type": "1", "metadata": {}, "hash": "89ef2d94085ed50fd42be18d4ab245f2f0020396f63f4ff0170e02295ec33e03", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 38965, "end_char_idx": 39888, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
❌         |      ❌      |
|              [Wav2Vec2Phoneme](model_doc/wav2vec2_phoneme)               |       ✅        |         ✅         |      ✅      |
|                         [WavLM](model_doc/wavlm)                         |       ✅        |         ❌         |      ❌      |
|                       [Whisper](model_doc/whisper)                       |       ✅        |         ✅         |      ✅      |
|                        [X-CLIP](model_doc/xclip)                         |       ✅        |         ❌         |      ❌      |
|                         [X-MOD](model_doc/xmod)                          |       ✅        |         ❌         |      ❌      |
|                          [XGLM](model_doc/xglm)                          |       ✅        |         ✅         |      ✅      |
|                           [XLM](model_doc/xlm)                           |       ✅        |         ✅         |      ❌      |
|


key: _node_content
string_value:
{"id_": "c2b5b845-88ed-4aea-8162-1c1c5545c969", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e206fb57-0267-45ba-9044-64ddb0490fa3", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "b1b7f99e1249cae4f67018422502eb66e5c2db4103100fe018479d38de2ea898", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9fa2af0d-f0fe-4ade-a091-dbcc415bb981", "node_type": "1", "metadata": {}, "hash": "04f00db1c44de05f1778bf3123d588e1811fd8dc898eab0707e316a69d6c39fc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 39450, "end_char_idx": 40384, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
|       ✅        |         ❌         |      ❌      |
|                         [X-MOD](model_doc/xmod)                          |       ✅        |         ❌         |      ❌      |
|                          [XGLM](model_doc/xglm)                          |       ✅        |         ✅         |      ✅      |
|                           [XLM](model_doc/xlm)                           |       ✅        |         ✅         |      ❌      |
|                [XLM-ProphetNet](model_doc/xlm-prophetnet)                |       ✅        |         ❌         |      ❌      |
|                   [XLM-RoBERTa](model_doc/xlm-roberta)                   |       ✅        |         ✅         |      ✅      |
|                [XLM-RoBERTa-XL](model_doc/xlm-roberta-xl)                |       ✅        |         ❌         |      ❌      |
|                         [XLM-V](model_doc/xlm-v)                         |       ✅        |         ✅         |


key: _node_content
string_value:
{"id_": "9fa2af0d-f0fe-4ade-a091-dbcc415bb981", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c2b5b845-88ed-4aea-8162-1c1c5545c969", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "b69f88ba6cd3fcffab8322a2e2ff41f7dc2f27bffd6abf97e6f298e784d18dfe", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "651dd095-7db7-49f8-a2d8-4a20cf3c9e9e", "node_type": "1", "metadata": {}, "hash": "35dcea3d60fa7250bcba97817c42fcda84183f4a83e6c35dfb96995870141bbf", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 39885, "end_char_idx": 40833, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
|
|                [XLM-ProphetNet](model_doc/xlm-prophetnet)                |       ✅        |         ❌         |      ❌      |
|                   [XLM-RoBERTa](model_doc/xlm-roberta)                   |       ✅        |         ✅         |      ✅      |
|                [XLM-RoBERTa-XL](model_doc/xlm-roberta-xl)                |       ✅        |         ❌         |      ❌      |
|                         [XLM-V](model_doc/xlm-v)                         |       ✅        |         ✅         |      ✅      |
|                         [XLNet](model_doc/xlnet)                         |       ✅        |         ✅         |      ❌      |
|                         [XLS-R](model_doc/xls_r)                         |       ✅        |         ✅         |      ✅      |
|                 [XLSR-Wav2Vec2](model_doc/xlsr_wav2vec2)                 |       ✅        |         ✅         |      ✅      |
|                         [YOLOS](model_doc/yolos)


key: _node_content
string_value:
{"id_": "651dd095-7db7-49f8-a2d8-4a20cf3c9e9e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ae56fe8d-ab49-4ab6-bff5-8b020af1916f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "836865c358dea9b81d80ebb705572b7f9f4dcfd6479dc208d646d8499b9bd468", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9fa2af0d-f0fe-4ade-a091-dbcc415bb981", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/index", "title": "\ud83e\udd17 Transformers", "tokens": 9875, "retrieve_doc": false, "source": "transformers"}, "hash": "37271e24208c4f7d72835757f0ae5d0644f8a6a4e79a3c9e235197ce04765f64", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 40383, "end_char_idx": 41185, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
|      ✅      |
|                         [XLNet](model_doc/xlnet)                         |       ✅        |         ✅         |      ❌      |
|                         [XLS-R](model_doc/xls_r)                         |       ✅        |         ✅         |      ✅      |
|                 [XLSR-Wav2Vec2](model_doc/xlsr_wav2vec2)                 |       ✅        |         ✅         |      ✅      |
|                         [YOLOS](model_doc/yolos)                         |       ✅        |         ❌         |      ❌      |
|                          [YOSO](model_doc/yoso)                          |       ✅        |         ❌         |      ❌      |
|                      [ZoeDepth](model_doc/zoedepth)                      |       ✅        |         ❌         |      ❌      |

<!-- End table-->


key: _node_content
string_value:
{"id_": "8e81a112-e222-44fc-83af-aaf1a27907de", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3d4a5be0-96a4-4cae-bb80-ec65cee94a22", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "bb1f2b699b61342ca13b75611df36eb9e3379ec4485edf0807dfbeb683246cf4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7a20c243-35c5-41a4-992b-0af2f7000f18", "node_type": "1", "metadata": {}, "hash": "18ceb50f5c015cfd9900fa1d273897dd22119d0eff75c60692928bbf7bd0485b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3282, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Optimizing LLMs for Speed and Memory

[[open-in-colab]]

Large Language Models (LLMs) such as GPT3/4, [Falcon](https://huggingface.co/tiiuae/falcon-40b), and [Llama](https://huggingface.co/meta-llama/Llama-2-70b-hf) are rapidly advancing in their ability to tackle human-centric tasks, establishing themselves as essential tools in modern knowledge-based industries.
Deploying these models in real-world tasks remains challenging, however:

-   To exhibit near-human text understanding and generation capabilities, LLMs currently require to be composed of billions of parameters (see [Kaplan et al](https://arxiv.org/abs/2001.08361), [Wei et. al](https://arxiv.org/abs/2206.07682)). This consequently amplifies the memory demands for inference.
-   In many real-world tasks, LLMs need to be given extensive contextual information. This necessitates the model's capability to manage very long input sequences during inference.

The crux of these challenges lies in augmenting the computational and memory capabilities of LLMs, especially when handling expansive input sequences.

In this guide, we will go over the effective techniques for efficient LLM deployment:

1.  **Lower Precision:** Research has shown that operating at reduced numerical precision, namely [8-bit and 4-bit](./main_classes/quantization.md) can achieve computational advantages without a considerable decline in model performance.

2.  **Flash Attention:** Flash Attention is a variation of the attention algorithm that not only provides a more memory-efficient approach but also realizes increased efficiency due to optimized GPU memory utilization.

3.  **Architectural Innovations:** Considering that LLMs are always deployed in the same way during inference, namely autoregressive text generation with a long input context, specialized model architectures have been proposed that allow for more efficient inference. The most important advancement in model architectures hereby are [Alibi](https://arxiv.org/abs/2108.12409), [Rotary embeddings](https://arxiv.org/abs/2104.09864), [Multi-Query Attention (MQA)](https://arxiv.org/abs/1911.02150) and [Grouped-Query-Attention (GQA)]((https://arxiv.org/abs/2305.13245)).

Throughout this guide, we will offer an analysis of auto-regressive generation from a tensor's perspective. We delve into the pros and cons of adopting lower precision, provide a comprehensive exploration of the latest attention algorithms, and discuss improved LLM architectures. While doing so, we run practical examples showcasing each of the feature improvements.

## 1. Lower Precision

Memory requirements of LLMs can be best understood by seeing the LLM as a set of weight matrices and vectors and the text inputs as a sequence of vectors. In the following, the definition *weights* will be used to signify all model weight matrices and vectors.

At the time of writing this guide, LLMs consist of at least a couple billion parameters. Each parameter thereby is made of a decimal number, e.g. `4.5689` which is usually stored in either [float32](https://en.wikipedia.org/wiki/Single-precision_floating-point_format), [bfloat16](https://en.wikipedia.org/wiki/Bfloat16_floating-point_format), or [float16](https://en.wikipedia.org/wiki/Half-precision_floating-point_format) format.


key: _node_content
string_value:
{"id_": "7a20c243-35c5-41a4-992b-0af2f7000f18", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3d4a5be0-96a4-4cae-bb80-ec65cee94a22", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "bb1f2b699b61342ca13b75611df36eb9e3379ec4485edf0807dfbeb683246cf4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8e81a112-e222-44fc-83af-aaf1a27907de", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "750b233959dd68b6e6cd300e2af360ac5516198f8be18e6c24ded286f9c2516b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9f18c838-0bda-4eff-96d8-9480047a1082", "node_type": "1", "metadata": {}, "hash": "ab1e062eedb3764ab287b71744c2fd9be9cb264aa71d1bca465816832d1ce640", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1627, "end_char_idx": 4102, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
3.  **Architectural Innovations:** Considering that LLMs are always deployed in the same way during inference, namely autoregressive text generation with a long input context, specialized model architectures have been proposed that allow for more efficient inference. The most important advancement in model architectures hereby are [Alibi](https://arxiv.org/abs/2108.12409), [Rotary embeddings](https://arxiv.org/abs/2104.09864), [Multi-Query Attention (MQA)](https://arxiv.org/abs/1911.02150) and [Grouped-Query-Attention (GQA)]((https://arxiv.org/abs/2305.13245)).

Throughout this guide, we will offer an analysis of auto-regressive generation from a tensor's perspective. We delve into the pros and cons of adopting lower precision, provide a comprehensive exploration of the latest attention algorithms, and discuss improved LLM architectures. While doing so, we run practical examples showcasing each of the feature improvements.

## 1. Lower Precision

Memory requirements of LLMs can be best understood by seeing the LLM as a set of weight matrices and vectors and the text inputs as a sequence of vectors. In the following, the definition *weights* will be used to signify all model weight matrices and vectors.

At the time of writing this guide, LLMs consist of at least a couple billion parameters. Each parameter thereby is made of a decimal number, e.g. `4.5689` which is usually stored in either [float32](https://en.wikipedia.org/wiki/Single-precision_floating-point_format), [bfloat16](https://en.wikipedia.org/wiki/Bfloat16_floating-point_format), or [float16](https://en.wikipedia.org/wiki/Half-precision_floating-point_format) format. This allows us to easily compute the memory requirement to load the LLM into memory:

> *Loading the weights of a model having X billion parameters requires roughly 4 * X GB of VRAM in float32 precision*

Nowadays, models are however rarely trained in full float32 precision, but usually in bfloat16 precision or less frequently in float16 precision. Therefore the rule of thumb becomes:

> *Loading the weights of a model having X billion parameters requires roughly 2 * X GB of VRAM in bfloat16/float16 precision*

For shorter text inputs (less than 1024 tokens), the memory requirement for inference is very much dominated by the memory requirement to load the weights. Therefore, for now, let's assume that the memory requirement for inference is equal to the memory requirement to load the model into the GPU VRAM.


key: _node_content
string_value:
{"id_": "9f18c838-0bda-4eff-96d8-9480047a1082", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3d4a5be0-96a4-4cae-bb80-ec65cee94a22", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "bb1f2b699b61342ca13b75611df36eb9e3379ec4485edf0807dfbeb683246cf4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7a20c243-35c5-41a4-992b-0af2f7000f18", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "3bd7fb37b2fd755752ef715ffa3716bf6c22bb61cfa7c7323fc3374c65f9b22f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b661d63f-cd67-459b-b0c4-c41d9f7a2737", "node_type": "1", "metadata": {}, "hash": "637e4164204c1370b39310c7856a390417c23a41d1275c376ef9cddb0ab3bc0f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2304, "end_char_idx": 5217, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
We delve into the pros and cons of adopting lower precision, provide a comprehensive exploration of the latest attention algorithms, and discuss improved LLM architectures. While doing so, we run practical examples showcasing each of the feature improvements.

## 1. Lower Precision

Memory requirements of LLMs can be best understood by seeing the LLM as a set of weight matrices and vectors and the text inputs as a sequence of vectors. In the following, the definition *weights* will be used to signify all model weight matrices and vectors.

At the time of writing this guide, LLMs consist of at least a couple billion parameters. Each parameter thereby is made of a decimal number, e.g. `4.5689` which is usually stored in either [float32](https://en.wikipedia.org/wiki/Single-precision_floating-point_format), [bfloat16](https://en.wikipedia.org/wiki/Bfloat16_floating-point_format), or [float16](https://en.wikipedia.org/wiki/Half-precision_floating-point_format) format. This allows us to easily compute the memory requirement to load the LLM into memory:

> *Loading the weights of a model having X billion parameters requires roughly 4 * X GB of VRAM in float32 precision*

Nowadays, models are however rarely trained in full float32 precision, but usually in bfloat16 precision or less frequently in float16 precision. Therefore the rule of thumb becomes:

> *Loading the weights of a model having X billion parameters requires roughly 2 * X GB of VRAM in bfloat16/float16 precision*

For shorter text inputs (less than 1024 tokens), the memory requirement for inference is very much dominated by the memory requirement to load the weights. Therefore, for now, let's assume that the memory requirement for inference is equal to the memory requirement to load the model into the GPU VRAM.

To give some examples of how much VRAM it roughly takes to load a model in bfloat16:

-   **GPT3** requires 2 \* 175 GB = **350 GB** VRAM
-   [**Bloom**](https://huggingface.co/bigscience/bloom) requires 2 \* 176 GB = **352 GB** VRAM
-   [**Llama-2-70b**](https://huggingface.co/meta-llama/Llama-2-70b-hf) requires 2 \* 70 GB = **140 GB** VRAM
-   [**Falcon-40b**](https://huggingface.co/tiiuae/falcon-40b) requires 2 \* 40 GB = **80 GB** VRAM
-   [**MPT-30b**](https://huggingface.co/mosaicml/mpt-30b) requires 2 \* 30 GB = **60 GB** VRAM
-   [**bigcode/starcoder**](https://huggingface.co/bigcode/starcoder) requires 2 \* 15.5 = **31 GB** VRAM

As of writing this document, the largest GPU chip on the market is the A100 & H100 offering 80GB of VRAM. Most of the models listed before require more than 80GB just to be loaded and therefore necessarily require [tensor parallelism](https://huggingface.co/docs/transformers/perf_train_gpu_many#tensor-parallelism) and/or [pipeline parallelism](https://huggingface.co/docs/transformers/perf_train_gpu_many#naive-model-parallelism-vertical-and-pipeline-parallelism).


key: _node_content
string_value:
{"id_": "b661d63f-cd67-459b-b0c4-c41d9f7a2737", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3d4a5be0-96a4-4cae-bb80-ec65cee94a22", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "bb1f2b699b61342ca13b75611df36eb9e3379ec4485edf0807dfbeb683246cf4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9f18c838-0bda-4eff-96d8-9480047a1082", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "3d35e9d9fe259ceebabe636c5237008d91cd27a4d9b8af703d0a11154cacd4ca", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e4af1050-bee8-4761-ab89-a3c7f598528c", "node_type": "1", "metadata": {}, "hash": "1787cf67b4e0c1f5f3ee54a0676f2ba0477b425237ca7298383638711980af20", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3956, "end_char_idx": 6790, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Therefore, for now, let's assume that the memory requirement for inference is equal to the memory requirement to load the model into the GPU VRAM.

To give some examples of how much VRAM it roughly takes to load a model in bfloat16:

-   **GPT3** requires 2 \* 175 GB = **350 GB** VRAM
-   [**Bloom**](https://huggingface.co/bigscience/bloom) requires 2 \* 176 GB = **352 GB** VRAM
-   [**Llama-2-70b**](https://huggingface.co/meta-llama/Llama-2-70b-hf) requires 2 \* 70 GB = **140 GB** VRAM
-   [**Falcon-40b**](https://huggingface.co/tiiuae/falcon-40b) requires 2 \* 40 GB = **80 GB** VRAM
-   [**MPT-30b**](https://huggingface.co/mosaicml/mpt-30b) requires 2 \* 30 GB = **60 GB** VRAM
-   [**bigcode/starcoder**](https://huggingface.co/bigcode/starcoder) requires 2 \* 15.5 = **31 GB** VRAM

As of writing this document, the largest GPU chip on the market is the A100 & H100 offering 80GB of VRAM. Most of the models listed before require more than 80GB just to be loaded and therefore necessarily require [tensor parallelism](https://huggingface.co/docs/transformers/perf_train_gpu_many#tensor-parallelism) and/or [pipeline parallelism](https://huggingface.co/docs/transformers/perf_train_gpu_many#naive-model-parallelism-vertical-and-pipeline-parallelism).

🤗 Transformers does not support tensor parallelism out of the box as it requires the model architecture to be written in a specific way. If you're interested in writing models in a tensor-parallelism-friendly way, feel free to have a look at [the text-generation-inference library](https://github.com/huggingface/text-generation-inference/tree/main/server/text_generation_server/models/custom_modeling).

Naive pipeline parallelism is supported out of the box. For this, simply load the model with `device="auto"` which will automatically place the different layers on the available GPUs as explained [here](https://huggingface.co/docs/accelerate/v0.22.0/en/concept_guides/big_model_inference).
Note, however that while very effective, this naive pipeline parallelism does not tackle the issues of GPU idling. For this more advanced pipeline parallelism is required as explained [here](https://huggingface.co/docs/transformers/en/perf_train_gpu_many#naive-model-parallelism-vertical-and-pipeline-parallelism).

If you have access to an 8 x 80GB A100 node, you could load BLOOM as follows

```bash
!pip install transformers accelerate bitsandbytes optimum
```
```python
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("bigscience/bloom", device_map="auto", pad_token_id=0)
```

By using `device_map="auto"` the attention layers would be equally distributed over all available GPUs.

In this guide, we will use [bigcode/octocoder](https://huggingface.co/bigcode/octocoder) as it can be run on a single 40 GB A100 GPU device chip.


key: _node_content
string_value:
{"id_": "e4af1050-bee8-4761-ab89-a3c7f598528c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3d4a5be0-96a4-4cae-bb80-ec65cee94a22", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "bb1f2b699b61342ca13b75611df36eb9e3379ec4485edf0807dfbeb683246cf4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b661d63f-cd67-459b-b0c4-c41d9f7a2737", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "5dd063ca12cf983af568f6fe7055aec40114375d9cd3b4c430dd2d6374ea0cc3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "63b4db71-af45-4fd1-97d3-ddcbef835ac5", "node_type": "1", "metadata": {}, "hash": "57697e8a545df8252585b85773501f9e061a33f3ba241660a58753937517cee3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5219, "end_char_idx": 8410, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
🤗 Transformers does not support tensor parallelism out of the box as it requires the model architecture to be written in a specific way. If you're interested in writing models in a tensor-parallelism-friendly way, feel free to have a look at [the text-generation-inference library](https://github.com/huggingface/text-generation-inference/tree/main/server/text_generation_server/models/custom_modeling).

Naive pipeline parallelism is supported out of the box. For this, simply load the model with `device="auto"` which will automatically place the different layers on the available GPUs as explained [here](https://huggingface.co/docs/accelerate/v0.22.0/en/concept_guides/big_model_inference).
Note, however that while very effective, this naive pipeline parallelism does not tackle the issues of GPU idling. For this more advanced pipeline parallelism is required as explained [here](https://huggingface.co/docs/transformers/en/perf_train_gpu_many#naive-model-parallelism-vertical-and-pipeline-parallelism).

If you have access to an 8 x 80GB A100 node, you could load BLOOM as follows

```bash
!pip install transformers accelerate bitsandbytes optimum
```
```python
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("bigscience/bloom", device_map="auto", pad_token_id=0)
```

By using `device_map="auto"` the attention layers would be equally distributed over all available GPUs.

In this guide, we will use [bigcode/octocoder](https://huggingface.co/bigcode/octocoder) as it can be run on a single 40 GB A100 GPU device chip. Note that all memory and speed optimizations that we will apply going forward, are equally applicable to models that require model or tensor parallelism.

Since the model is loaded in bfloat16 precision, using our rule of thumb above, we would expect the memory requirement to run inference with `bigcode/octocoder` to be around 31 GB VRAM. Let's give it a try.

We first load the model and tokenizer and then pass both to Transformers' [pipeline](https://huggingface.co/docs/transformers/main_classes/pipelines) object.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
import torch

model = AutoModelForCausalLM.from_pretrained("bigcode/octocoder", torch_dtype=torch.bfloat16, device_map="auto", pad_token_id=0)
tokenizer = AutoTokenizer.from_pretrained("bigcode/octocoder")

pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)
```

```python
prompt = "Question: Please write a function in Python that transforms bytes to Giga bytes.\n\nAnswer:"

result = pipe(prompt, max_new_tokens=60)[0]["generated_text"][len(prompt):]
result
```

**Output**:
```
Here is a Python function that transforms bytes to Giga bytes:\n\n```python\ndef bytes_to_giga_bytes(bytes):\n    return bytes / 1024 / 1024 / 1024\n```\n\nThis function takes a single
```

Nice, we can now directly use the result to convert bytes into Gigabytes.

```python
def bytes_to_giga_bytes(bytes):
  return bytes / 1024 / 1024 / 1024
```

Let's call [`torch.cuda.max_memory_allocated`](https://pytorch.org/docs/stable/generated/torch.cuda.max_memory_allocated.html) to measure the peak GPU memory allocation.


key: _node_content
string_value:
{"id_": "63b4db71-af45-4fd1-97d3-ddcbef835ac5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3d4a5be0-96a4-4cae-bb80-ec65cee94a22", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "bb1f2b699b61342ca13b75611df36eb9e3379ec4485edf0807dfbeb683246cf4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e4af1050-bee8-4761-ab89-a3c7f598528c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "4194f31cb28ee16c87609b4c3b86218cff423d8551c816443049ef8ed94726b2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "48fbda2c-e4b7-49d2-be11-7d69427b7086", "node_type": "1", "metadata": {}, "hash": "43e5700112c8da992cf7535e6b834d154270f07660268e9926241bcb0928af88", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6791, "end_char_idx": 9901, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Note that all memory and speed optimizations that we will apply going forward, are equally applicable to models that require model or tensor parallelism.

Since the model is loaded in bfloat16 precision, using our rule of thumb above, we would expect the memory requirement to run inference with `bigcode/octocoder` to be around 31 GB VRAM. Let's give it a try.

We first load the model and tokenizer and then pass both to Transformers' [pipeline](https://huggingface.co/docs/transformers/main_classes/pipelines) object.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
import torch

model = AutoModelForCausalLM.from_pretrained("bigcode/octocoder", torch_dtype=torch.bfloat16, device_map="auto", pad_token_id=0)
tokenizer = AutoTokenizer.from_pretrained("bigcode/octocoder")

pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)
```

```python
prompt = "Question: Please write a function in Python that transforms bytes to Giga bytes.\n\nAnswer:"

result = pipe(prompt, max_new_tokens=60)[0]["generated_text"][len(prompt):]
result
```

**Output**:
```
Here is a Python function that transforms bytes to Giga bytes:\n\n```python\ndef bytes_to_giga_bytes(bytes):\n    return bytes / 1024 / 1024 / 1024\n```\n\nThis function takes a single
```

Nice, we can now directly use the result to convert bytes into Gigabytes.

```python
def bytes_to_giga_bytes(bytes):
  return bytes / 1024 / 1024 / 1024
```

Let's call [`torch.cuda.max_memory_allocated`](https://pytorch.org/docs/stable/generated/torch.cuda.max_memory_allocated.html) to measure the peak GPU memory allocation.

```python
bytes_to_giga_bytes(torch.cuda.max_memory_allocated())
```

**Output**:
```bash
29.0260648727417
```

Close enough to our back-of-the-envelope computation! We can see the number is not exactly correct as going from bytes to kilobytes requires a multiplication of 1024 instead of 1000. Therefore the back-of-the-envelope formula can also be understood as an "at most X GB" computation.
Note that if we had tried to run the model in full float32 precision, a whopping 64 GB of VRAM would have been required.

> Almost all models are trained in bfloat16 nowadays, there is no reason to run the model in full float32 precision if [your GPU supports bfloat16](https://discuss.pytorch.org/t/bfloat16-native-support/117155/5). Float32 won't give better inference results than the precision that was used to train the model.

If you are unsure in which format the model weights are stored on the Hub, you can always look into the checkpoint's config under `"torch_dtype"`, *e.g.* [here](https://huggingface.co/meta-llama/Llama-2-7b-hf/blob/6fdf2e60f86ff2481f2241aaee459f85b5b0bbb9/config.json#L21). It is recommended to set the model to the same precision type as written in the config when loading with `from_pretrained(..., torch_dtype=...)` except when the original type is float32 in which case one can use both `float16` or `bfloat16` for inference.


Let's define a `flush(...)` function to free all allocated memory so that we can accurately measure the peak allocated GPU memory.


key: _node_content
string_value:
{"id_": "48fbda2c-e4b7-49d2-be11-7d69427b7086", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3d4a5be0-96a4-4cae-bb80-ec65cee94a22", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "bb1f2b699b61342ca13b75611df36eb9e3379ec4485edf0807dfbeb683246cf4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "63b4db71-af45-4fd1-97d3-ddcbef835ac5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "35bb99dc3ae88cc831d4a2a9758311c6b7f960a9c6f1fc36d4a7e33d79a218d7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "670b0cf2-f9e5-4203-a400-0876d14f369b", "node_type": "1", "metadata": {}, "hash": "a00581045b45484befbe08bca38772d4c8c0ca1d032bc477874c6ef0a12216d0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8412, "end_char_idx": 11493, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
bytes_to_giga_bytes(torch.cuda.max_memory_allocated())
```

**Output**:
```bash
29.0260648727417
```

Close enough to our back-of-the-envelope computation! We can see the number is not exactly correct as going from bytes to kilobytes requires a multiplication of 1024 instead of 1000. Therefore the back-of-the-envelope formula can also be understood as an "at most X GB" computation.
Note that if we had tried to run the model in full float32 precision, a whopping 64 GB of VRAM would have been required.

> Almost all models are trained in bfloat16 nowadays, there is no reason to run the model in full float32 precision if [your GPU supports bfloat16](https://discuss.pytorch.org/t/bfloat16-native-support/117155/5). Float32 won't give better inference results than the precision that was used to train the model.

If you are unsure in which format the model weights are stored on the Hub, you can always look into the checkpoint's config under `"torch_dtype"`, *e.g.* [here](https://huggingface.co/meta-llama/Llama-2-7b-hf/blob/6fdf2e60f86ff2481f2241aaee459f85b5b0bbb9/config.json#L21). It is recommended to set the model to the same precision type as written in the config when loading with `from_pretrained(..., torch_dtype=...)` except when the original type is float32 in which case one can use both `float16` or `bfloat16` for inference.


Let's define a `flush(...)` function to free all allocated memory so that we can accurately measure the peak allocated GPU memory.

```python
del pipe
del model

import gc
import torch

def flush():
  gc.collect()
  torch.cuda.empty_cache()
  torch.cuda.reset_peak_memory_stats()
```

Let's call it now for the next experiment.

```python
flush()
```
In the recent version of the accelerate library, you can also use a utility method called `release_memory()`

```python
from accelerate.utils import release_memory
# ...

release_memory(model)
```

Now what if your GPU does not have 32 GB of VRAM? It has been found that model weights can be quantized to 8-bit or 4-bits without a significant loss in performance (see [Dettmers et al.](https://arxiv.org/abs/2208.07339)).
Model can be quantized to even 3 or 2 bits with an acceptable loss in performance as shown in the recent [GPTQ paper](https://arxiv.org/abs/2210.17323) 🤯.

Without going into too many details, quantization schemes aim at reducing the precision of weights while trying to keep the model's inference results as accurate as possible (*a.k.a* as close as possible to bfloat16).
Note that quantization works especially well for text generation since all we care about is choosing the *set of most likely next tokens* and don't really care about the exact values of the next token *logit* distribution.
All that matters is that the next token *logit* distribution stays roughly the same so that an `argmax` or `topk` operation gives the same results.

There are various quantization techniques, which we won't discuss in detail here, but in general, all quantization techniques work as follows:

-   1.  Quantize all weights to the target precision
-   2.


key: _node_content
string_value:
{"id_": "670b0cf2-f9e5-4203-a400-0876d14f369b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3d4a5be0-96a4-4cae-bb80-ec65cee94a22", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "bb1f2b699b61342ca13b75611df36eb9e3379ec4485edf0807dfbeb683246cf4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "48fbda2c-e4b7-49d2-be11-7d69427b7086", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "9c9cddc3d0e15a0f081499f8383301c5672eae164622a6e1286b53b7ee4b0553", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bfe74bf8-afab-4812-8bd4-93f44f9c5adf", "node_type": "1", "metadata": {}, "hash": "d47e75e1347e6f704d3fa7fbec990d7cfadca43f436a05fe4534ed088a686baf", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9903, "end_char_idx": 12742, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
del pipe
del model

import gc
import torch

def flush():
  gc.collect()
  torch.cuda.empty_cache()
  torch.cuda.reset_peak_memory_stats()
```

Let's call it now for the next experiment.

```python
flush()
```
In the recent version of the accelerate library, you can also use a utility method called `release_memory()`

```python
from accelerate.utils import release_memory
# ...

release_memory(model)
```

Now what if your GPU does not have 32 GB of VRAM? It has been found that model weights can be quantized to 8-bit or 4-bits without a significant loss in performance (see [Dettmers et al.](https://arxiv.org/abs/2208.07339)).
Model can be quantized to even 3 or 2 bits with an acceptable loss in performance as shown in the recent [GPTQ paper](https://arxiv.org/abs/2210.17323) 🤯.

Without going into too many details, quantization schemes aim at reducing the precision of weights while trying to keep the model's inference results as accurate as possible (*a.k.a* as close as possible to bfloat16).
Note that quantization works especially well for text generation since all we care about is choosing the *set of most likely next tokens* and don't really care about the exact values of the next token *logit* distribution.
All that matters is that the next token *logit* distribution stays roughly the same so that an `argmax` or `topk` operation gives the same results.

There are various quantization techniques, which we won't discuss in detail here, but in general, all quantization techniques work as follows:

-   1.  Quantize all weights to the target precision
-   2.  Load the quantized weights, and pass the input sequence of vectors in bfloat16 precision
-   3.  Dynamically dequantize weights to bfloat16 to perform the computation with their input vectors in bfloat16 precision

In a nutshell, this means that *inputs-weight matrix* multiplications, with \\( X \\) being the *inputs*, \\( W \\) being a weight matrix and \\( Y \\) being the output:

$$ Y = X * W $$

are changed to

$$ Y = X * \text{dequantize}(W) $$

for every matrix multiplication. Dequantization and re-quantization is performed sequentially for all weight matrices as the inputs run through the network graph.

Therefore, inference time is often **not** reduced when using quantized weights, but rather increases.
Enough theory, let's give it a try! To quantize the weights with Transformers, you need to make sure that
the [`bitsandbytes`](https://github.com/TimDettmers/bitsandbytes) library is installed.

```bash
!pip install bitsandbytes
```

We can then load models in 8-bit quantization by simply adding a `load_in_8bit=True` flag to `from_pretrained`.

```python
model = AutoModelForCausalLM.from_pretrained("bigcode/octocoder", load_in_8bit=True, pad_token_id=0)
```

Now, let's run our example again and measure the memory usage.


key: _node_content
string_value:
{"id_": "bfe74bf8-afab-4812-8bd4-93f44f9c5adf", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3d4a5be0-96a4-4cae-bb80-ec65cee94a22", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "bb1f2b699b61342ca13b75611df36eb9e3379ec4485edf0807dfbeb683246cf4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "670b0cf2-f9e5-4203-a400-0876d14f369b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "f0c796e46edb85f2c38d6577a68436d8056ff4063bdcee9274a1696beac27a3f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "dcbdb020-1df7-4596-b9c6-1fdfc11f17e0", "node_type": "1", "metadata": {}, "hash": "bb9222685e59f39a7662ae5f59923f59ca4c6175c8cf66bcbe0c912178ebdbea", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11141, "end_char_idx": 13966, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
All that matters is that the next token *logit* distribution stays roughly the same so that an `argmax` or `topk` operation gives the same results.

There are various quantization techniques, which we won't discuss in detail here, but in general, all quantization techniques work as follows:

-   1.  Quantize all weights to the target precision
-   2.  Load the quantized weights, and pass the input sequence of vectors in bfloat16 precision
-   3.  Dynamically dequantize weights to bfloat16 to perform the computation with their input vectors in bfloat16 precision

In a nutshell, this means that *inputs-weight matrix* multiplications, with \\( X \\) being the *inputs*, \\( W \\) being a weight matrix and \\( Y \\) being the output:

$$ Y = X * W $$

are changed to

$$ Y = X * \text{dequantize}(W) $$

for every matrix multiplication. Dequantization and re-quantization is performed sequentially for all weight matrices as the inputs run through the network graph.

Therefore, inference time is often **not** reduced when using quantized weights, but rather increases.
Enough theory, let's give it a try! To quantize the weights with Transformers, you need to make sure that
the [`bitsandbytes`](https://github.com/TimDettmers/bitsandbytes) library is installed.

```bash
!pip install bitsandbytes
```

We can then load models in 8-bit quantization by simply adding a `load_in_8bit=True` flag to `from_pretrained`.

```python
model = AutoModelForCausalLM.from_pretrained("bigcode/octocoder", load_in_8bit=True, pad_token_id=0)
```

Now, let's run our example again and measure the memory usage.

```python
pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)

result = pipe(prompt, max_new_tokens=60)[0]["generated_text"][len(prompt):]
result
```

**Output**:
```
Here is a Python function that transforms bytes to Giga bytes:\n\n```python\ndef bytes_to_giga_bytes(bytes):\n    return bytes / 1024 / 1024 / 1024\n```\n\nThis function takes a single
```

Nice, we're getting the same result as before, so no loss in accuracy! Let's look at how much memory was used this time.

```python
bytes_to_giga_bytes(torch.cuda.max_memory_allocated())
```

**Output**:
```
15.219234466552734
```

Significantly less! We're down to just a bit over 15 GBs and could therefore run this model on consumer GPUs like the 4090.
We're seeing a very nice gain in memory efficiency and more or less no degradation to the model's output. However, we can also notice a slight slow-down during inference.


We delete the models and flush the memory again.
```python
del model
del pipe
```

```python
flush()
```

Let's see what peak GPU memory consumption 4-bit quantization gives. Quantizing the model to 4-bit can be done with the same API as before - this time by passing `load_in_4bit=True` instead of `load_in_8bit=True`.


key: _node_content
string_value:
{"id_": "dcbdb020-1df7-4596-b9c6-1fdfc11f17e0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3d4a5be0-96a4-4cae-bb80-ec65cee94a22", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "bb1f2b699b61342ca13b75611df36eb9e3379ec4485edf0807dfbeb683246cf4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bfe74bf8-afab-4812-8bd4-93f44f9c5adf", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "aa8cd3be093046d3257fbfab84458a41573d1754dd9c01f708a8c3e29c21e395", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "34c71458-216c-4008-ba1a-b44ba8573d82", "node_type": "1", "metadata": {}, "hash": "2bd5ee5757e14d9009141febf29deb688a2c282eda6159678f8c696d279cdcba", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12564, "end_char_idx": 15302, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
model = AutoModelForCausalLM.from_pretrained("bigcode/octocoder", load_in_8bit=True, pad_token_id=0)
```

Now, let's run our example again and measure the memory usage.

```python
pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)

result = pipe(prompt, max_new_tokens=60)[0]["generated_text"][len(prompt):]
result
```

**Output**:
```
Here is a Python function that transforms bytes to Giga bytes:\n\n```python\ndef bytes_to_giga_bytes(bytes):\n    return bytes / 1024 / 1024 / 1024\n```\n\nThis function takes a single
```

Nice, we're getting the same result as before, so no loss in accuracy! Let's look at how much memory was used this time.

```python
bytes_to_giga_bytes(torch.cuda.max_memory_allocated())
```

**Output**:
```
15.219234466552734
```

Significantly less! We're down to just a bit over 15 GBs and could therefore run this model on consumer GPUs like the 4090.
We're seeing a very nice gain in memory efficiency and more or less no degradation to the model's output. However, we can also notice a slight slow-down during inference.


We delete the models and flush the memory again.
```python
del model
del pipe
```

```python
flush()
```

Let's see what peak GPU memory consumption 4-bit quantization gives. Quantizing the model to 4-bit can be done with the same API as before - this time by passing `load_in_4bit=True` instead of `load_in_8bit=True`.

```python
model = AutoModelForCausalLM.from_pretrained("bigcode/octocoder", load_in_4bit=True, low_cpu_mem_usage=True, pad_token_id=0)

pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)

result = pipe(prompt, max_new_tokens=60)[0]["generated_text"][len(prompt):]
result
```

**Output**:
```
Here is a Python function that transforms bytes to Giga bytes:\n\n```\ndef bytes_to_gigabytes(bytes):\n    return bytes / 1024 / 1024 / 1024\n```\n\nThis function takes a single argument
```

We're almost seeing the same output text as before - just the `python` is missing just before the code snippet. Let's see how much memory was required.

```python
bytes_to_giga_bytes(torch.cuda.max_memory_allocated())
```

**Output**:
```
9.543574333190918
```

Just 9.5GB! That's really not a lot for a >15 billion parameter model.

While we see very little degradation in accuracy for our model here, 4-bit quantization can in practice often lead to different results compared to 8-bit quantization or full `bfloat16` inference. It is up to the user to try it out.

Also note that inference here was again a bit slower compared to 8-bit quantization which is due to the more aggressive quantization method used for 4-bit quantization leading to \\( \text{quantize} \\) and \\( \text{dequantize} \\) taking longer during inference.


key: _node_content
string_value:
{"id_": "34c71458-216c-4008-ba1a-b44ba8573d82", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3d4a5be0-96a4-4cae-bb80-ec65cee94a22", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "bb1f2b699b61342ca13b75611df36eb9e3379ec4485edf0807dfbeb683246cf4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "dcbdb020-1df7-4596-b9c6-1fdfc11f17e0", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "c5212e80043bf9a429e65b0354ba865733d9ecb61838d049bcfee67541c126c2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "cacd0fe4-95f3-48ad-98e6-b0f1f6c6ab71", "node_type": "1", "metadata": {}, "hash": "98304f94ad2869bf3228b38524be068ad7f14e0ca8a602f28ad77d8e601015bd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13822, "end_char_idx": 16979, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Quantizing the model to 4-bit can be done with the same API as before - this time by passing `load_in_4bit=True` instead of `load_in_8bit=True`.

```python
model = AutoModelForCausalLM.from_pretrained("bigcode/octocoder", load_in_4bit=True, low_cpu_mem_usage=True, pad_token_id=0)

pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)

result = pipe(prompt, max_new_tokens=60)[0]["generated_text"][len(prompt):]
result
```

**Output**:
```
Here is a Python function that transforms bytes to Giga bytes:\n\n```\ndef bytes_to_gigabytes(bytes):\n    return bytes / 1024 / 1024 / 1024\n```\n\nThis function takes a single argument
```

We're almost seeing the same output text as before - just the `python` is missing just before the code snippet. Let's see how much memory was required.

```python
bytes_to_giga_bytes(torch.cuda.max_memory_allocated())
```

**Output**:
```
9.543574333190918
```

Just 9.5GB! That's really not a lot for a >15 billion parameter model.

While we see very little degradation in accuracy for our model here, 4-bit quantization can in practice often lead to different results compared to 8-bit quantization or full `bfloat16` inference. It is up to the user to try it out.

Also note that inference here was again a bit slower compared to 8-bit quantization which is due to the more aggressive quantization method used for 4-bit quantization leading to \\( \text{quantize} \\) and \\( \text{dequantize} \\) taking longer during inference.

```python
del model
del pipe
```
```python
flush()
```

Overall, we saw that running OctoCoder in 8-bit precision reduced the required GPU VRAM from 32G GPU VRAM to only 15GB and running the model in 4-bit precision further reduces the required GPU VRAM to just a bit over 9GB.

4-bit quantization allows the model to be run on GPUs such as RTX3090, V100, and T4 which are quite accessible for most people.

For more information on quantization and to see how one can quantize models to require even less GPU VRAM memory than 4-bit, we recommend looking into the [`AutoGPTQ`](https://huggingface.co/docs/transformers/main/en/main_classes/quantization#autogptq-integration%60) implementation.

> As a conclusion, it is important to remember that model quantization trades improved memory efficiency against accuracy and in some cases inference time.

If GPU memory is not a constraint for your use case, there is often no need to look into quantization. However many GPUs simply can't run LLMs without quantization methods and in this case, 4-bit and 8-bit quantization schemes are extremely useful tools.

For more in-detail usage information, we strongly recommend taking a look at the [Transformers Quantization Docs](https://huggingface.co/docs/transformers/main_classes/quantization#general-usage).
Next, let's look into how we can improve computational and memory efficiency by using better algorithms and an improved model architecture.

## 2. Flash Attention

Today's top-performing LLMs share more or less the same fundamental architecture that consists of feed-forward layers, activation layers, layer normalization layers, and most crucially, self-attention layers.


key: _node_content
string_value:
{"id_": "cacd0fe4-95f3-48ad-98e6-b0f1f6c6ab71", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3d4a5be0-96a4-4cae-bb80-ec65cee94a22", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "bb1f2b699b61342ca13b75611df36eb9e3379ec4485edf0807dfbeb683246cf4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "34c71458-216c-4008-ba1a-b44ba8573d82", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "22264f169ad35254910c57ac855bb5cd23d205510c68d050670526b697139af4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "44f679f5-557d-4203-ab68-02a6559a04a5", "node_type": "1", "metadata": {}, "hash": "023f51058026d1f146a53e2218eef553bab246f4dd0130463d98121450872a77", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15304, "end_char_idx": 18322, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
del model
del pipe
```
```python
flush()
```

Overall, we saw that running OctoCoder in 8-bit precision reduced the required GPU VRAM from 32G GPU VRAM to only 15GB and running the model in 4-bit precision further reduces the required GPU VRAM to just a bit over 9GB.

4-bit quantization allows the model to be run on GPUs such as RTX3090, V100, and T4 which are quite accessible for most people.

For more information on quantization and to see how one can quantize models to require even less GPU VRAM memory than 4-bit, we recommend looking into the [`AutoGPTQ`](https://huggingface.co/docs/transformers/main/en/main_classes/quantization#autogptq-integration%60) implementation.

> As a conclusion, it is important to remember that model quantization trades improved memory efficiency against accuracy and in some cases inference time.

If GPU memory is not a constraint for your use case, there is often no need to look into quantization. However many GPUs simply can't run LLMs without quantization methods and in this case, 4-bit and 8-bit quantization schemes are extremely useful tools.

For more in-detail usage information, we strongly recommend taking a look at the [Transformers Quantization Docs](https://huggingface.co/docs/transformers/main_classes/quantization#general-usage).
Next, let's look into how we can improve computational and memory efficiency by using better algorithms and an improved model architecture.

## 2. Flash Attention

Today's top-performing LLMs share more or less the same fundamental architecture that consists of feed-forward layers, activation layers, layer normalization layers, and most crucially, self-attention layers.

Self-attention layers are central to Large Language Models (LLMs) in that they enable the model to understand the contextual relationships between input tokens.
However, the peak GPU memory consumption for self-attention layers grows *quadratically* both in compute and memory complexity with number of input tokens (also called *sequence length*) that we denote in the following by \\( N \\) .
While this is not really noticeable for shorter input sequences (of up to 1000 input tokens), it becomes a serious problem for longer input sequences (at around 16000 input tokens).

Let's take a closer look. The formula to compute the output \\( \mathbf{O} \\) of a self-attention layer for an input \\( \mathbf{X} \\) of length \\( N \\) is:

$$ \textbf{O} = \text{Attn}(\mathbf{X}) = \mathbf{V} \times \text{Softmax}(\mathbf{QK}^T) \text{ with } \mathbf{Q} = \mathbf{W}_q \mathbf{X}, \mathbf{V} = \mathbf{W}_v \mathbf{X}, \mathbf{K} = \mathbf{W}_k \mathbf{X} $$

\\(  \mathbf{X} = (\mathbf{x}_1, ... \mathbf{x}_{N}) \\) is thereby the input sequence to the attention layer. The projections \\( \mathbf{Q} \\) and \\( \mathbf{K} \\) will each consist of \\( N \\) vectors resulting in the \\( \mathbf{QK}^T \\) being of size \\( N^2 \\) .

LLMs usually have multiple attention heads, thus doing multiple self-attention computations in parallel.


key: _node_content
string_value:
{"id_": "44f679f5-557d-4203-ab68-02a6559a04a5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3d4a5be0-96a4-4cae-bb80-ec65cee94a22", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "bb1f2b699b61342ca13b75611df36eb9e3379ec4485edf0807dfbeb683246cf4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "cacd0fe4-95f3-48ad-98e6-b0f1f6c6ab71", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "895e88f176bc2c2ba17b91a376639974bd07efa22186ccddab5c419744cfa648", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1c529f0a-781f-436d-9199-3f9cd24c1576", "node_type": "1", "metadata": {}, "hash": "da13bc0ab8fec21fc3182e9f565ba4eac443cc461ee00d3688e877450f2e616c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16981, "end_char_idx": 19439, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Self-attention layers are central to Large Language Models (LLMs) in that they enable the model to understand the contextual relationships between input tokens.
However, the peak GPU memory consumption for self-attention layers grows *quadratically* both in compute and memory complexity with number of input tokens (also called *sequence length*) that we denote in the following by \\( N \\) .
While this is not really noticeable for shorter input sequences (of up to 1000 input tokens), it becomes a serious problem for longer input sequences (at around 16000 input tokens).

Let's take a closer look. The formula to compute the output \\( \mathbf{O} \\) of a self-attention layer for an input \\( \mathbf{X} \\) of length \\( N \\) is:

$$ \textbf{O} = \text{Attn}(\mathbf{X}) = \mathbf{V} \times \text{Softmax}(\mathbf{QK}^T) \text{ with } \mathbf{Q} = \mathbf{W}_q \mathbf{X}, \mathbf{V} = \mathbf{W}_v \mathbf{X}, \mathbf{K} = \mathbf{W}_k \mathbf{X} $$

\\(  \mathbf{X} = (\mathbf{x}_1, ... \mathbf{x}_{N}) \\) is thereby the input sequence to the attention layer. The projections \\( \mathbf{Q} \\) and \\( \mathbf{K} \\) will each consist of \\( N \\) vectors resulting in the \\( \mathbf{QK}^T \\) being of size \\( N^2 \\) .

LLMs usually have multiple attention heads, thus doing multiple self-attention computations in parallel.
Assuming, the LLM has 40 attention heads and runs in bfloat16 precision, we can calculate the memory requirement to store the \\( \mathbf{QK^T} \\) matrices to be \\( 40 * 2 * N^2 \\) bytes. For \\( N=1000 \\) only around 50 MB of VRAM are needed, however, for \\( N=16000 \\) we would need 19 GB of VRAM, and for \\( N=100,000 \\) we would need almost 1TB just to store the \\( \mathbf{QK}^T \\) matrices.

Long story short, the default self-attention algorithm quickly becomes prohibitively memory-expensive for large input contexts.

As LLMs improve in text comprehension and generation, they are applied to increasingly complex tasks. While models once handled the translation or summarization of a few sentences, they now manage entire pages, demanding the capability to process extensive input lengths.

How can we get rid of the exorbitant memory requirements for large input lengths? We need a new way to compute the self-attention mechanism that gets rid of the \\( QK^T \\) matrix. [Tri Dao et al.](https://arxiv.org/abs/2205.14135) developed exactly such a new algorithm and called it **Flash Attention**.


key: _node_content
string_value:
{"id_": "1c529f0a-781f-436d-9199-3f9cd24c1576", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3d4a5be0-96a4-4cae-bb80-ec65cee94a22", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "bb1f2b699b61342ca13b75611df36eb9e3379ec4485edf0807dfbeb683246cf4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "44f679f5-557d-4203-ab68-02a6559a04a5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "f19f4f6cd670f843a2b1174af47de89fbcb980425e74b79ba49cc58b79f514be", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "759094c4-a051-4136-8639-35850ed14617", "node_type": "1", "metadata": {}, "hash": "60d9ca93c754891cb6388c141a3d9001b9f04f822dabc8d8bb6f27db750c8ba3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18053, "end_char_idx": 20731, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The projections \\( \mathbf{Q} \\) and \\( \mathbf{K} \\) will each consist of \\( N \\) vectors resulting in the \\( \mathbf{QK}^T \\) being of size \\( N^2 \\) .

LLMs usually have multiple attention heads, thus doing multiple self-attention computations in parallel.
Assuming, the LLM has 40 attention heads and runs in bfloat16 precision, we can calculate the memory requirement to store the \\( \mathbf{QK^T} \\) matrices to be \\( 40 * 2 * N^2 \\) bytes. For \\( N=1000 \\) only around 50 MB of VRAM are needed, however, for \\( N=16000 \\) we would need 19 GB of VRAM, and for \\( N=100,000 \\) we would need almost 1TB just to store the \\( \mathbf{QK}^T \\) matrices.

Long story short, the default self-attention algorithm quickly becomes prohibitively memory-expensive for large input contexts.

As LLMs improve in text comprehension and generation, they are applied to increasingly complex tasks. While models once handled the translation or summarization of a few sentences, they now manage entire pages, demanding the capability to process extensive input lengths.

How can we get rid of the exorbitant memory requirements for large input lengths? We need a new way to compute the self-attention mechanism that gets rid of the \\( QK^T \\) matrix. [Tri Dao et al.](https://arxiv.org/abs/2205.14135) developed exactly such a new algorithm and called it **Flash Attention**.

In a nutshell, Flash Attention breaks the  \\(\mathbf{V} \times \text{Softmax}(\mathbf{QK}^T\\)) computation apart and instead computes smaller chunks of the output by iterating over multiple softmax computation steps:

$$ \textbf{O}_i \leftarrow s^a_{ij} * \textbf{O}_i + s^b_{ij} * \mathbf{V}_{j} \times \text{Softmax}(\mathbf{QK}^T_{i,j}) \text{ for multiple } i, j \text{ iterations} $$

with \\( s^a_{ij} \\) and \\( s^b_{ij} \\) being some softmax normalization statistics that need to be recomputed for every \\( i \\) and \\( j \\) .

Please note that the whole Flash Attention is a bit more complex and is greatly simplified here as going in too much depth is out of scope for this guide. The reader is invited to take a look at the well-written [Flash Attention paper](https://arxiv.org/abs/2205.14135) for more details.

The main takeaway here is:

> By keeping track of softmax normalization statistics and by using some smart mathematics, Flash Attention gives **numerical identical** outputs compared to the default self-attention layer at a memory cost that only increases linearly with \\( N \\) .

Looking at the formula, one would intuitively say that Flash Attention must be much slower compared to the default self-attention formula as more computation needs to be done.


key: _node_content
string_value:
{"id_": "759094c4-a051-4136-8639-35850ed14617", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3d4a5be0-96a4-4cae-bb80-ec65cee94a22", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "bb1f2b699b61342ca13b75611df36eb9e3379ec4485edf0807dfbeb683246cf4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1c529f0a-781f-436d-9199-3f9cd24c1576", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "a4d5c9b165609075480926898cc436827b420c25abe52ac72fffc7b88e979314", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0c197c04-e552-4f3a-98a7-692a030fab82", "node_type": "1", "metadata": {}, "hash": "bee6b6b65bed6cfa10cf27d5134d37246194f246e884428eb00cd4bd5581cd03", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19215, "end_char_idx": 22592, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
We need a new way to compute the self-attention mechanism that gets rid of the \\( QK^T \\) matrix. [Tri Dao et al.](https://arxiv.org/abs/2205.14135) developed exactly such a new algorithm and called it **Flash Attention**.

In a nutshell, Flash Attention breaks the  \\(\mathbf{V} \times \text{Softmax}(\mathbf{QK}^T\\)) computation apart and instead computes smaller chunks of the output by iterating over multiple softmax computation steps:

$$ \textbf{O}_i \leftarrow s^a_{ij} * \textbf{O}_i + s^b_{ij} * \mathbf{V}_{j} \times \text{Softmax}(\mathbf{QK}^T_{i,j}) \text{ for multiple } i, j \text{ iterations} $$

with \\( s^a_{ij} \\) and \\( s^b_{ij} \\) being some softmax normalization statistics that need to be recomputed for every \\( i \\) and \\( j \\) .

Please note that the whole Flash Attention is a bit more complex and is greatly simplified here as going in too much depth is out of scope for this guide. The reader is invited to take a look at the well-written [Flash Attention paper](https://arxiv.org/abs/2205.14135) for more details.

The main takeaway here is:

> By keeping track of softmax normalization statistics and by using some smart mathematics, Flash Attention gives **numerical identical** outputs compared to the default self-attention layer at a memory cost that only increases linearly with \\( N \\) .

Looking at the formula, one would intuitively say that Flash Attention must be much slower compared to the default self-attention formula as more computation needs to be done. Indeed Flash Attention requires more FLOPs compared to normal attention as the softmax normalization statistics have to constantly be recomputed (see [paper](https://arxiv.org/abs/2205.14135) for more details if interested)

> However, Flash Attention is much faster in inference compared to default attention which comes from its ability to significantly reduce the demands on the slower, high-bandwidth memory of the GPU (VRAM), focusing instead on the faster on-chip memory (SRAM).

Essentially, Flash Attention makes sure that all intermediate write and read operations can be done using the fast *on-chip* SRAM memory instead of having to access the slower VRAM memory to compute the output vector \\( \mathbf{O} \\) .

In practice, there is currently absolutely no reason to **not** use Flash Attention if available. The algorithm gives mathematically the same outputs, and is both faster and more memory-efficient.

Let's look at a practical example.

Our OctoCoder model now gets a significantly longer input prompt which includes a so-called *system prompt*. System prompts are used to steer the LLM into a better assistant that is tailored to the users' task.
In the following, we use a system prompt that will make OctoCoder a better coding assistant.

```python
system_prompt = """Below are a series of dialogues between various people and an AI technical assistant.
The assistant tries to be helpful, polite, honest, sophisticated, emotionally aware, and humble but knowledgeable.
The assistant is happy to help with code questions and will do their best to understand exactly what is needed.
It also tries to avoid giving false or misleading information, and it caveats when it isn't entirely sure about the right answer.
That said, the assistant is practical really does its best, and doesn't let caution get too much in the way of being useful.


key: _node_content
string_value:
{"id_": "0c197c04-e552-4f3a-98a7-692a030fab82", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3d4a5be0-96a4-4cae-bb80-ec65cee94a22", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "bb1f2b699b61342ca13b75611df36eb9e3379ec4485edf0807dfbeb683246cf4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "759094c4-a051-4136-8639-35850ed14617", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "539beb3591c95a9ffd59e66f3fe28a79fcfd35e235da0ca0c4b8034aa3925af2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a92e8777-bcd9-45ff-ac19-49360bca21b4", "node_type": "1", "metadata": {}, "hash": "ec6de3ec1a73968158524336b615c00b29a5022500cefa565d80500b76674616", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 20732, "end_char_idx": 24177, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Indeed Flash Attention requires more FLOPs compared to normal attention as the softmax normalization statistics have to constantly be recomputed (see [paper](https://arxiv.org/abs/2205.14135) for more details if interested)

> However, Flash Attention is much faster in inference compared to default attention which comes from its ability to significantly reduce the demands on the slower, high-bandwidth memory of the GPU (VRAM), focusing instead on the faster on-chip memory (SRAM).

Essentially, Flash Attention makes sure that all intermediate write and read operations can be done using the fast *on-chip* SRAM memory instead of having to access the slower VRAM memory to compute the output vector \\( \mathbf{O} \\) .

In practice, there is currently absolutely no reason to **not** use Flash Attention if available. The algorithm gives mathematically the same outputs, and is both faster and more memory-efficient.

Let's look at a practical example.

Our OctoCoder model now gets a significantly longer input prompt which includes a so-called *system prompt*. System prompts are used to steer the LLM into a better assistant that is tailored to the users' task.
In the following, we use a system prompt that will make OctoCoder a better coding assistant.

```python
system_prompt = """Below are a series of dialogues between various people and an AI technical assistant.
The assistant tries to be helpful, polite, honest, sophisticated, emotionally aware, and humble but knowledgeable.
The assistant is happy to help with code questions and will do their best to understand exactly what is needed.
It also tries to avoid giving false or misleading information, and it caveats when it isn't entirely sure about the right answer.
That said, the assistant is practical really does its best, and doesn't let caution get too much in the way of being useful.

The Starcoder models are a series of 15.5B parameter models trained on 80+ programming languages from The Stack (v1.2) (excluding opt-out requests).
The model uses Multi Query Attention, was trained using the Fill-in-the-Middle objective, and with 8,192 tokens context window for a trillion tokens of heavily deduplicated data.

-----

Question: Write a function that takes two lists and returns a list that has alternating elements from each input list.

Answer: Sure. Here is a function that does that.

def alternating(list1, list2):
   results = []
   for i in range(len(list1)):
       results.append(list1[i])
       results.append(list2[i])
   return results

Question: Can you write some test cases for this function?

Answer: Sure, here are some tests.

assert alternating([10, 20, 30], [1, 2, 3]) == [10, 1, 20, 2, 30, 3]
assert alternating([True, False], [4, 5]) == [True, 4, False, 5]
assert alternating([], []) == []

Question: Modify the function so that it returns all input elements when the lists have uneven length. The elements from the longer list should be at the end.

Answer: Here is the modified function.

def alternating(list1, list2):
   results = []
   for i in range(min(len(list1), len(list2))):
       results.append(list1[i])
       results.append(list2[i])
   if len(list1) > len(list2):
       results.extend(list1[i+1:])
   else:
       results.extend(list2[i+1:])
   return results

-----
"""
```
For demonstration purposes, we duplicate the system prompt by ten so that the input length is long enough to observe Flash Attention's memory savings.


key: _node_content
string_value:
{"id_": "a92e8777-bcd9-45ff-ac19-49360bca21b4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3d4a5be0-96a4-4cae-bb80-ec65cee94a22", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "bb1f2b699b61342ca13b75611df36eb9e3379ec4485edf0807dfbeb683246cf4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0c197c04-e552-4f3a-98a7-692a030fab82", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "56dcc3a11332a9bb16f33c53d76ee1a3be87b1ee0e070dcbe12b8928917d6602", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1dfe62c1-9482-4ae1-9612-be8aeb08402f", "node_type": "1", "metadata": {}, "hash": "81edcc7eb0791a113b33f41742762d92cfdabb0372dad5da735ab96b7ee72a8d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 22594, "end_char_idx": 25689, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The Starcoder models are a series of 15.5B parameter models trained on 80+ programming languages from The Stack (v1.2) (excluding opt-out requests).
The model uses Multi Query Attention, was trained using the Fill-in-the-Middle objective, and with 8,192 tokens context window for a trillion tokens of heavily deduplicated data.

-----

Question: Write a function that takes two lists and returns a list that has alternating elements from each input list.

Answer: Sure. Here is a function that does that.

def alternating(list1, list2):
   results = []
   for i in range(len(list1)):
       results.append(list1[i])
       results.append(list2[i])
   return results

Question: Can you write some test cases for this function?

Answer: Sure, here are some tests.

assert alternating([10, 20, 30], [1, 2, 3]) == [10, 1, 20, 2, 30, 3]
assert alternating([True, False], [4, 5]) == [True, 4, False, 5]
assert alternating([], []) == []

Question: Modify the function so that it returns all input elements when the lists have uneven length. The elements from the longer list should be at the end.

Answer: Here is the modified function.

def alternating(list1, list2):
   results = []
   for i in range(min(len(list1), len(list2))):
       results.append(list1[i])
       results.append(list2[i])
   if len(list1) > len(list2):
       results.extend(list1[i+1:])
   else:
       results.extend(list2[i+1:])
   return results

-----
"""
```
For demonstration purposes, we duplicate the system prompt by ten so that the input length is long enough to observe Flash Attention's memory savings.
We append the original text prompt `"Question: Please write a function in Python that transforms bytes to Giga bytes.\n\nAnswer: Here"`

```python
long_prompt = 10 * system_prompt + prompt
```

We instantiate our model again in bfloat16 precision.

```python
model = AutoModelForCausalLM.from_pretrained("bigcode/octocoder", torch_dtype=torch.bfloat16, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("bigcode/octocoder")

pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)
```

Let's now run the model just like before *without Flash Attention* and measure the peak GPU memory requirement and inference time.

```python
import time

start_time = time.time()
result = pipe(long_prompt, max_new_tokens=60)[0]["generated_text"][len(long_prompt):]

print(f"Generated in {time.time() - start_time} seconds.")
result
```

**Output**:
```
Generated in 10.96854019165039 seconds.
Sure. Here is a function that does that.\n\ndef bytes_to_giga(bytes):\n   return bytes / 1024 / 1024 / 1024\n\nAnswer: Sure. Here is a function that does that.\n\ndef
````

We're getting the same output as before, however this time, the model repeats the answer multiple times until it's 60 tokens cut-off. This is not surprising as we've repeated the system prompt ten times for demonstration purposes and thus cued the model to repeat itself.

**Note** that the system prompt should not be repeated ten times in real-world applications - one time is enough!

Let's measure the peak GPU memory requirement.


key: _node_content
string_value:
{"id_": "1dfe62c1-9482-4ae1-9612-be8aeb08402f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3d4a5be0-96a4-4cae-bb80-ec65cee94a22", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "bb1f2b699b61342ca13b75611df36eb9e3379ec4485edf0807dfbeb683246cf4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a92e8777-bcd9-45ff-ac19-49360bca21b4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "e2dc162f6c3175b6bbbea27979c64ac239b9d255c65e1609f8f63686d9e82cff", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "47937dea-7a4b-4dd2-b291-35bcee6a12a6", "node_type": "1", "metadata": {}, "hash": "ae05df733e7b8251b12aca7853fb9d1b226908defa724bcdf9b88e95dcd21560", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 24178, "end_char_idx": 27345, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
We append the original text prompt `"Question: Please write a function in Python that transforms bytes to Giga bytes.\n\nAnswer: Here"`

```python
long_prompt = 10 * system_prompt + prompt
```

We instantiate our model again in bfloat16 precision.

```python
model = AutoModelForCausalLM.from_pretrained("bigcode/octocoder", torch_dtype=torch.bfloat16, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("bigcode/octocoder")

pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)
```

Let's now run the model just like before *without Flash Attention* and measure the peak GPU memory requirement and inference time.

```python
import time

start_time = time.time()
result = pipe(long_prompt, max_new_tokens=60)[0]["generated_text"][len(long_prompt):]

print(f"Generated in {time.time() - start_time} seconds.")
result
```

**Output**:
```
Generated in 10.96854019165039 seconds.
Sure. Here is a function that does that.\n\ndef bytes_to_giga(bytes):\n   return bytes / 1024 / 1024 / 1024\n\nAnswer: Sure. Here is a function that does that.\n\ndef
````

We're getting the same output as before, however this time, the model repeats the answer multiple times until it's 60 tokens cut-off. This is not surprising as we've repeated the system prompt ten times for demonstration purposes and thus cued the model to repeat itself.

**Note** that the system prompt should not be repeated ten times in real-world applications - one time is enough!

Let's measure the peak GPU memory requirement.

```python
bytes_to_giga_bytes(torch.cuda.max_memory_allocated())
```

**Output**:
```bash
37.668193340301514
```

As we can see the peak GPU memory requirement is now significantly higher than in the beginning, which is largely due to the longer input sequence. Also the generation takes a little over a minute now.

We call `flush()` to free GPU memory for our next experiment.

```python
flush()
```

For comparison, let's run the same function, but enable Flash Attention instead.
To do so, we convert the model to [BetterTransformer](https://huggingface.co/docs/optimum/bettertransformer/overview) and by doing so enabling PyTorch's [SDPA self-attention](https://pytorch.org/docs/master/generated/torch.nn.functional.scaled_dot_product_attention) which in turn is able to use Flash Attention.

```python
model.to_bettertransformer()
```

Now we run the exact same code snippet as before and under the hood Transformers will make use of Flash Attention.

```py
start_time = time.time()
with torch.backends.cuda.sdp_kernel(enable_flash=True, enable_math=False, enable_mem_efficient=False):
    result = pipe(long_prompt, max_new_tokens=60)[0]["generated_text"][len(long_prompt):]

print(f"Generated in {time.time() - start_time} seconds.")
result
```

**Output**:
```
Generated in 3.0211617946624756 seconds.
 Sure. Here is a function that does that.\n\ndef bytes_to_giga(bytes):\n   return bytes / 1024 / 1024 / 1024\n\nAnswer: Sure. Here is a function that does that.\n\ndef
```

We're getting the exact same result as before, but can observe a very significant speed-up thanks to Flash Attention.

Let's measure the memory consumption one last time.


key: _node_content
string_value:
{"id_": "47937dea-7a4b-4dd2-b291-35bcee6a12a6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3d4a5be0-96a4-4cae-bb80-ec65cee94a22", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "bb1f2b699b61342ca13b75611df36eb9e3379ec4485edf0807dfbeb683246cf4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1dfe62c1-9482-4ae1-9612-be8aeb08402f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "516dcb65e982cb09387d7fec9e080d13c16955a625b857c20fe5dbdbcbe0943c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "da94834b-58a3-4012-9e3c-4c0bc27a124b", "node_type": "1", "metadata": {}, "hash": "b174175882ca439a6bb2ab7a212e4785ef187849050afcede8f6e684397ef1f9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 25691, "end_char_idx": 28965, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
bytes_to_giga_bytes(torch.cuda.max_memory_allocated())
```

**Output**:
```bash
37.668193340301514
```

As we can see the peak GPU memory requirement is now significantly higher than in the beginning, which is largely due to the longer input sequence. Also the generation takes a little over a minute now.

We call `flush()` to free GPU memory for our next experiment.

```python
flush()
```

For comparison, let's run the same function, but enable Flash Attention instead.
To do so, we convert the model to [BetterTransformer](https://huggingface.co/docs/optimum/bettertransformer/overview) and by doing so enabling PyTorch's [SDPA self-attention](https://pytorch.org/docs/master/generated/torch.nn.functional.scaled_dot_product_attention) which in turn is able to use Flash Attention.

```python
model.to_bettertransformer()
```

Now we run the exact same code snippet as before and under the hood Transformers will make use of Flash Attention.

```py
start_time = time.time()
with torch.backends.cuda.sdp_kernel(enable_flash=True, enable_math=False, enable_mem_efficient=False):
    result = pipe(long_prompt, max_new_tokens=60)[0]["generated_text"][len(long_prompt):]

print(f"Generated in {time.time() - start_time} seconds.")
result
```

**Output**:
```
Generated in 3.0211617946624756 seconds.
 Sure. Here is a function that does that.\n\ndef bytes_to_giga(bytes):\n   return bytes / 1024 / 1024 / 1024\n\nAnswer: Sure. Here is a function that does that.\n\ndef
```

We're getting the exact same result as before, but can observe a very significant speed-up thanks to Flash Attention.

Let's measure the memory consumption one last time.

```python
bytes_to_giga_bytes(torch.cuda.max_memory_allocated())
```

**Output**:
```
32.617331981658936
```

And we're almost back to our original 29GB peak GPU memory from the beginning.

We can observe that we only use roughly 100MB more GPU memory when passing a very long input sequence with Flash Attention compared to passing a short input sequence as done in the beginning.

```py
flush()
```

For more information on how to use Flash Attention, please have a look at [this doc page](https://huggingface.co/docs/transformers/en/perf_infer_gpu_one#flashattention-2).

## 3. Architectural Innovations

So far we have looked into improving computational and memory efficiency by:

-   Casting the weights to a lower precision format
-   Replacing the self-attention algorithm with a more memory- and compute efficient version

Let's now look into how we can change the architecture of an LLM so that it is most effective and efficient for task that require long text inputs, *e.g.*:
-   Retrieval augmented Questions Answering,
-   Summarization,
-   Chat

Note that *chat* not only requires the LLM to handle long text inputs, but it also necessitates that the LLM is able to efficiently handle the back-and-forth dialogue between user and assistant (such as ChatGPT).

Once trained, the fundamental LLM architecture is difficult to change, so it is important to make considerations about the LLM's tasks beforehand and accordingly optimize the model's architecture.
There are two important components of the model architecture that quickly become memory and/or performance bottlenecks for large input sequences.


key: _node_content
string_value:
{"id_": "da94834b-58a3-4012-9e3c-4c0bc27a124b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3d4a5be0-96a4-4cae-bb80-ec65cee94a22", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "bb1f2b699b61342ca13b75611df36eb9e3379ec4485edf0807dfbeb683246cf4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "47937dea-7a4b-4dd2-b291-35bcee6a12a6", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "982dbe0f6752637dfc883d12a69a94173c1fb9e7c95125dd31c52b0783e7d652", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e28d1f2a-4d3f-4c6f-ac22-a64f858c157f", "node_type": "1", "metadata": {}, "hash": "f477813cd5adf852190df5bb9c246dceac104b413cba1347853ab0228dc94175", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 27294, "end_char_idx": 30558, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Let's measure the memory consumption one last time.

```python
bytes_to_giga_bytes(torch.cuda.max_memory_allocated())
```

**Output**:
```
32.617331981658936
```

And we're almost back to our original 29GB peak GPU memory from the beginning.

We can observe that we only use roughly 100MB more GPU memory when passing a very long input sequence with Flash Attention compared to passing a short input sequence as done in the beginning.

```py
flush()
```

For more information on how to use Flash Attention, please have a look at [this doc page](https://huggingface.co/docs/transformers/en/perf_infer_gpu_one#flashattention-2).

## 3. Architectural Innovations

So far we have looked into improving computational and memory efficiency by:

-   Casting the weights to a lower precision format
-   Replacing the self-attention algorithm with a more memory- and compute efficient version

Let's now look into how we can change the architecture of an LLM so that it is most effective and efficient for task that require long text inputs, *e.g.*:
-   Retrieval augmented Questions Answering,
-   Summarization,
-   Chat

Note that *chat* not only requires the LLM to handle long text inputs, but it also necessitates that the LLM is able to efficiently handle the back-and-forth dialogue between user and assistant (such as ChatGPT).

Once trained, the fundamental LLM architecture is difficult to change, so it is important to make considerations about the LLM's tasks beforehand and accordingly optimize the model's architecture.
There are two important components of the model architecture that quickly become memory and/or performance bottlenecks for large input sequences.

-   The positional embeddings
-   The key-value cache

Let's go over each component in more detail

### 3.1 Improving positional embeddings of LLMs

Self-attention puts each token in relation to each other's tokens.
As an example, the \\( \text{Softmax}(\mathbf{QK}^T) \\) matrix of the text input sequence *"Hello", "I", "love", "you"* could look as follows:

![](/blog/assets/163_optimize_llm/self_attn_tokens.png)

Each word token is given a probability mass at which it attends all other word tokens and, therefore is put into relation with all other word tokens. E.g. the word *"love"* attends to the word *"Hello"* with 5%, to *"I"* with 30%, and to itself with 65%.

A LLM based on self-attention, but without position embeddings would have great difficulties in understanding the positions of the text inputs to each other.
This is because the probability score computed by \\( \mathbf{QK}^T \\) relates each word token to each other word token in \\( O(1) \\) computations regardless of their relative positional distance to each other.
Therefore, for the LLM without position embeddings each token appears to have the same distance to all other tokens, *e.g.* differentiating between *"Hello I love you"* and *"You love I hello"* would be very challenging.

For the LLM to understand sentence order, an additional *cue* is needed and is usually applied in the form of *positional encodings* (or also called *positional embeddings*).
Positional encodings, encode the position of each token into a numerical presentation that the LLM can leverage to better understand sentence order.


key: _node_content
string_value:
{"id_": "e28d1f2a-4d3f-4c6f-ac22-a64f858c157f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3d4a5be0-96a4-4cae-bb80-ec65cee94a22", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "bb1f2b699b61342ca13b75611df36eb9e3379ec4485edf0807dfbeb683246cf4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "da94834b-58a3-4012-9e3c-4c0bc27a124b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "4972cd960e5ee06ee1a1ee8b555a1f61d84f0c9f11b2116a91767b009f09f243", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "15b7e771-c820-4034-961b-972c3b0f5fbf", "node_type": "1", "metadata": {}, "hash": "73f2fbbd3c80bae45682a4f88bd216b152d3d9ed185b7a7bf34dec9608ecbb3d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 28820, "end_char_idx": 31803, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
There are two important components of the model architecture that quickly become memory and/or performance bottlenecks for large input sequences.

-   The positional embeddings
-   The key-value cache

Let's go over each component in more detail

### 3.1 Improving positional embeddings of LLMs

Self-attention puts each token in relation to each other's tokens.
As an example, the \\( \text{Softmax}(\mathbf{QK}^T) \\) matrix of the text input sequence *"Hello", "I", "love", "you"* could look as follows:

![](/blog/assets/163_optimize_llm/self_attn_tokens.png)

Each word token is given a probability mass at which it attends all other word tokens and, therefore is put into relation with all other word tokens. E.g. the word *"love"* attends to the word *"Hello"* with 5%, to *"I"* with 30%, and to itself with 65%.

A LLM based on self-attention, but without position embeddings would have great difficulties in understanding the positions of the text inputs to each other.
This is because the probability score computed by \\( \mathbf{QK}^T \\) relates each word token to each other word token in \\( O(1) \\) computations regardless of their relative positional distance to each other.
Therefore, for the LLM without position embeddings each token appears to have the same distance to all other tokens, *e.g.* differentiating between *"Hello I love you"* and *"You love I hello"* would be very challenging.

For the LLM to understand sentence order, an additional *cue* is needed and is usually applied in the form of *positional encodings* (or also called *positional embeddings*).
Positional encodings, encode the position of each token into a numerical presentation that the LLM can leverage to better understand sentence order.

The authors of the [*Attention Is All You Need*](https://arxiv.org/abs/1706.03762) paper introduced sinusoidal positional embeddings \\( \mathbf{P} = \mathbf{p}_1, \ldots, \mathbf{p}_N \\) .
where each vector \\( \mathbf{p}_i \\) is computed as a sinusoidal function of its position \\( i \\) .
The positional encodings are then simply added to the input sequence vectors \\( \mathbf{\hat{X}} = \mathbf{\hat{x}}_1, \ldots, \mathbf{\hat{x}}_N \\) = \\( \mathbf{x}_1 + \mathbf{p}_1, \ldots, \mathbf{x}_N + \mathbf{p}_N \\) thereby cueing the model to better learn sentence order.

Instead of using fixed position embeddings, others (such as [Devlin et al.](https://arxiv.org/abs/1810.04805)) used learned positional encodings for which the positional embeddings
\\( \mathbf{P} \\) are learned during training.

Sinusoidal and learned position embeddings used to be the predominant methods to encode sentence order into LLMs, but a couple of problems related to these positional encodings were found:

  1. Sinusoidal and learned position embeddings are both absolute positional embeddings, *i.e.* encoding a unique embedding for each position id: \\( 0, \ldots, N \\) . As shown by [Huang et al.](https://arxiv.org/abs/2009.13658) and [Su et al.


key: _node_content
string_value:
{"id_": "15b7e771-c820-4034-961b-972c3b0f5fbf", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3d4a5be0-96a4-4cae-bb80-ec65cee94a22", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "bb1f2b699b61342ca13b75611df36eb9e3379ec4485edf0807dfbeb683246cf4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e28d1f2a-4d3f-4c6f-ac22-a64f858c157f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "0bbb3af180c1b66140bc6401011fbd10e38d9188f025e3b229129462a2505e0a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7968093d-b602-4b6c-b50b-b6cef32d8f5c", "node_type": "1", "metadata": {}, "hash": "3534575b8eb919461581803b041d68e76b87488bd34a642843ae49317445727d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 30410, "end_char_idx": 33251, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Positional encodings, encode the position of each token into a numerical presentation that the LLM can leverage to better understand sentence order.

The authors of the [*Attention Is All You Need*](https://arxiv.org/abs/1706.03762) paper introduced sinusoidal positional embeddings \\( \mathbf{P} = \mathbf{p}_1, \ldots, \mathbf{p}_N \\) .
where each vector \\( \mathbf{p}_i \\) is computed as a sinusoidal function of its position \\( i \\) .
The positional encodings are then simply added to the input sequence vectors \\( \mathbf{\hat{X}} = \mathbf{\hat{x}}_1, \ldots, \mathbf{\hat{x}}_N \\) = \\( \mathbf{x}_1 + \mathbf{p}_1, \ldots, \mathbf{x}_N + \mathbf{p}_N \\) thereby cueing the model to better learn sentence order.

Instead of using fixed position embeddings, others (such as [Devlin et al.](https://arxiv.org/abs/1810.04805)) used learned positional encodings for which the positional embeddings
\\( \mathbf{P} \\) are learned during training.

Sinusoidal and learned position embeddings used to be the predominant methods to encode sentence order into LLMs, but a couple of problems related to these positional encodings were found:

  1. Sinusoidal and learned position embeddings are both absolute positional embeddings, *i.e.* encoding a unique embedding for each position id: \\( 0, \ldots, N \\) . As shown by [Huang et al.](https://arxiv.org/abs/2009.13658) and [Su et al.](https://arxiv.org/abs/2104.09864), absolute positional embeddings lead to poor LLM performance for long text inputs. For long text inputs, it is advantageous if the model learns the relative positional distance input tokens have to each other instead of their absolute position.
  2. When using learned position embeddings, the LLM has to be trained on a fixed input length \\( N \\), which makes it difficult to extrapolate to an input length longer than what it was trained on.

Recently, relative positional embeddings that can tackle the above mentioned problems have become more popular, most notably:

-   [Rotary Position Embedding (RoPE)](https://arxiv.org/abs/2104.09864)
-   [ALiBi](https://arxiv.org/abs/2108.12409)

Both *RoPE* and *ALiBi* argue that it's best to cue the LLM about sentence order directly in the self-attention algorithm as it's there that word tokens are put into relation with each other. More specifically, sentence order should be cued by modifying the \\( \mathbf{QK}^T \\) computation.

Without going into too many details, *RoPE* notes that positional information can be encoded into query-key pairs, *e.g.* \\( \mathbf{q}_i \\) and \\( \mathbf{x}_j \\) by rotating each vector by an angle \\( \theta * i \\) and \\( \theta * j \\) respectively with \\( i, j \\) describing each vectors sentence position:

$$ \mathbf{\hat{q}}_i^T \mathbf{\hat{x}}_j = \mathbf{{q}}_i^T \mathbf{R}_{\theta, i -j} \mathbf{{x}}_j.


key: _node_content
string_value:
{"id_": "7968093d-b602-4b6c-b50b-b6cef32d8f5c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3d4a5be0-96a4-4cae-bb80-ec65cee94a22", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "bb1f2b699b61342ca13b75611df36eb9e3379ec4485edf0807dfbeb683246cf4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "15b7e771-c820-4034-961b-972c3b0f5fbf", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "4c0be1b0be39510ddcf4ebda86b3b0967ebcbfbdbded9fd8c4bd175cc349dc9e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fe84e6bb-664e-48ee-a3ee-197ba5633947", "node_type": "1", "metadata": {}, "hash": "746622e56f12dd42c4e15c33214f92ccb751c5c3fde74580d9681e1527b3f1f7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 31803, "end_char_idx": 34520, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
](https://arxiv.org/abs/2104.09864), absolute positional embeddings lead to poor LLM performance for long text inputs. For long text inputs, it is advantageous if the model learns the relative positional distance input tokens have to each other instead of their absolute position.
  2. When using learned position embeddings, the LLM has to be trained on a fixed input length \\( N \\), which makes it difficult to extrapolate to an input length longer than what it was trained on.

Recently, relative positional embeddings that can tackle the above mentioned problems have become more popular, most notably:

-   [Rotary Position Embedding (RoPE)](https://arxiv.org/abs/2104.09864)
-   [ALiBi](https://arxiv.org/abs/2108.12409)

Both *RoPE* and *ALiBi* argue that it's best to cue the LLM about sentence order directly in the self-attention algorithm as it's there that word tokens are put into relation with each other. More specifically, sentence order should be cued by modifying the \\( \mathbf{QK}^T \\) computation.

Without going into too many details, *RoPE* notes that positional information can be encoded into query-key pairs, *e.g.* \\( \mathbf{q}_i \\) and \\( \mathbf{x}_j \\) by rotating each vector by an angle \\( \theta * i \\) and \\( \theta * j \\) respectively with \\( i, j \\) describing each vectors sentence position:

$$ \mathbf{\hat{q}}_i^T \mathbf{\hat{x}}_j = \mathbf{{q}}_i^T \mathbf{R}_{\theta, i -j} \mathbf{{x}}_j. $$

\\( \mathbf{R}_{\theta, i - j} \\) thereby represents a rotational matrix. \\( \theta \\) is *not* learned during training, but instead set to a pre-defined value that depends on the maximum input sequence length during training.

> By doing so, the propability score between \\( \mathbf{q}_i \\) and \\( \mathbf{q}_j \\) is only affected if \\( i \ne j \\) and solely depends on the relative distance \\( i - j \\) regardless of each vector's specific positions \\( i \\) and \\( j \\) .

*RoPE* is used in multiple of today's most important LLMs, such as:

-   [**Falcon**](https://huggingface.co/tiiuae/falcon-40b)
-   [**Llama**](https://arxiv.org/abs/2302.13971)
-   [**PaLM**](https://arxiv.org/abs/2204.02311)

As an alternative, *ALiBi* proposes a much simpler relative position encoding scheme. The relative distance that input tokens have to each other is added as a negative integer scaled by a pre-defined value `m` to each query-key entry of the \\( \mathbf{QK}^T \\) matrix right before the softmax computation.

![](/blog/assets/163_optimize_llm/alibi.png)

As shown in the [ALiBi](https://arxiv.org/abs/2108.12409) paper, this simple relative positional encoding allows the model to retain a high performance even at very long text input sequences.


key: _node_content
string_value:
{"id_": "fe84e6bb-664e-48ee-a3ee-197ba5633947", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3d4a5be0-96a4-4cae-bb80-ec65cee94a22", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "bb1f2b699b61342ca13b75611df36eb9e3379ec4485edf0807dfbeb683246cf4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7968093d-b602-4b6c-b50b-b6cef32d8f5c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "8c4ccb196e159e1ae787a982818e9084a673e8bfb8fd44b1bbfa8192f7473351", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "57150321-dfaa-4fe3-a076-ae59685a4468", "node_type": "1", "metadata": {}, "hash": "952fb74b52be8619a9ba89b8c64a0e4e109a868436509abc357a1b1466675bc8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 33252, "end_char_idx": 36207, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
$$

\\( \mathbf{R}_{\theta, i - j} \\) thereby represents a rotational matrix. \\( \theta \\) is *not* learned during training, but instead set to a pre-defined value that depends on the maximum input sequence length during training.

> By doing so, the propability score between \\( \mathbf{q}_i \\) and \\( \mathbf{q}_j \\) is only affected if \\( i \ne j \\) and solely depends on the relative distance \\( i - j \\) regardless of each vector's specific positions \\( i \\) and \\( j \\) .

*RoPE* is used in multiple of today's most important LLMs, such as:

-   [**Falcon**](https://huggingface.co/tiiuae/falcon-40b)
-   [**Llama**](https://arxiv.org/abs/2302.13971)
-   [**PaLM**](https://arxiv.org/abs/2204.02311)

As an alternative, *ALiBi* proposes a much simpler relative position encoding scheme. The relative distance that input tokens have to each other is added as a negative integer scaled by a pre-defined value `m` to each query-key entry of the \\( \mathbf{QK}^T \\) matrix right before the softmax computation.

![](/blog/assets/163_optimize_llm/alibi.png)

As shown in the [ALiBi](https://arxiv.org/abs/2108.12409) paper, this simple relative positional encoding allows the model to retain a high performance even at very long text input sequences.

*ALiBi* is used in multiple of today's most important LLMs, such as:

-   [**MPT**](https://huggingface.co/mosaicml/mpt-30b)
-   [**BLOOM**](https://huggingface.co/bigscience/bloom)

Both *RoPE* and *ALiBi* position encodings can extrapolate to input lengths not seen during training whereas it has been shown that extrapolation works much better out-of-the-box for *ALiBi* as compared to *RoPE*.
For ALiBi, one simply increases the values of the lower triangular position matrix to match the length of the input sequence.
For *RoPE*, keeping the same \\( \theta \\) that was used during training leads to poor results when passing text inputs much longer than those seen during training, *c.f* [Press et al.](https://arxiv.org/abs/2108.12409). However, the community has found a couple of effective tricks that adapt \\( \theta \\), thereby allowing *RoPE* position embeddings to work well for extrapolated text input sequences (see [here](https://github.com/huggingface/transformers/pull/24653)).

> Both RoPE and ALiBi are relative positional embeddings that are *not* learned during training, but instead are based on the following intuitions:
 -   Positional cues about the text inputs should be given directly to the \\( QK^T \\) matrix of the self-attention layer
 -   The LLM should be incentivized to learn a constant *relative* distance positional encodings have to each other
 -   The further text input tokens are from each other, the lower the probability of their query-value probability. Both RoPE and ALiBi lower the query-key probability of tokens far away from each other. RoPE by decreasing their vector product by increasing the angle between the query-key vectors.


key: _node_content
string_value:
{"id_": "57150321-dfaa-4fe3-a076-ae59685a4468", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3d4a5be0-96a4-4cae-bb80-ec65cee94a22", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "bb1f2b699b61342ca13b75611df36eb9e3379ec4485edf0807dfbeb683246cf4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fe84e6bb-664e-48ee-a3ee-197ba5633947", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "be2dbae258c9cdadc18ebcbc10075e5365ece61122b351f4dab0bff429a0e5da", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d4e0cca1-e525-4759-b8ad-bf2a82c5dec1", "node_type": "1", "metadata": {}, "hash": "24846d182f5654940a979efdbf2dff9f3fe461faabaf22c36dcce381893c0799", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 34919, "end_char_idx": 38101, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
For ALiBi, one simply increases the values of the lower triangular position matrix to match the length of the input sequence.
For *RoPE*, keeping the same \\( \theta \\) that was used during training leads to poor results when passing text inputs much longer than those seen during training, *c.f* [Press et al.](https://arxiv.org/abs/2108.12409). However, the community has found a couple of effective tricks that adapt \\( \theta \\), thereby allowing *RoPE* position embeddings to work well for extrapolated text input sequences (see [here](https://github.com/huggingface/transformers/pull/24653)).

> Both RoPE and ALiBi are relative positional embeddings that are *not* learned during training, but instead are based on the following intuitions:
 -   Positional cues about the text inputs should be given directly to the \\( QK^T \\) matrix of the self-attention layer
 -   The LLM should be incentivized to learn a constant *relative* distance positional encodings have to each other
 -   The further text input tokens are from each other, the lower the probability of their query-value probability. Both RoPE and ALiBi lower the query-key probability of tokens far away from each other. RoPE by decreasing their vector product by increasing the angle between the query-key vectors. ALiBi by adding large negative numbers to the vector product

In conclusion, LLMs that are intended to be deployed in tasks that require handling large text inputs are better trained with relative positional embeddings, such as RoPE and ALiBi. Also note that even if an LLM with RoPE and ALiBi has been trained only on a fixed length of say \\( N_1 = 2048 \\) it can still be used in practice with text inputs much larger than \\( N_1 \\), like \\( N_2 = 8192 > N_1 \\) by extrapolating the positional embeddings.

### 3.2 The key-value cache

Auto-regressive text generation with LLMs works by iteratively putting in an input sequence, sampling the next token, appending the next token to the input sequence, and continuing to do so until the LLM produces a token that signifies that the generation has finished.

Please have a look at [Transformer's Generate Text Tutorial](https://huggingface.co/docs/transformers/llm_tutorial#generate-text) to get a more visual explanation of how auto-regressive generation works.

Let's run a quick code snippet to show how auto-regressive works in practice. We will simply take the most likely next token via `torch.argmax`.

```python
input_ids = tokenizer(prompt, return_tensors="pt")["input_ids"].to("cuda")

for _ in range(5):
  next_logits = model(input_ids)["logits"][:, -1:]
  next_token_id = torch.argmax(next_logits,dim=-1)

  input_ids = torch.cat([input_ids, next_token_id], dim=-1)
  print("shape of input_ids", input_ids.shape)

generated_text = tokenizer.batch_decode(input_ids[:, -5:])
generated_text
```

**Output**:
```
shape of input_ids torch.Size([1, 21])
shape of input_ids torch.Size([1, 22])
shape of input_ids torch.Size([1, 23])
shape of input_ids torch.Size([1, 24])
shape of input_ids torch.Size([1, 25])
[' Here is a Python function']
```

As we can see every time we increase the text input tokens by the just sampled token.


key: _node_content
string_value:
{"id_": "d4e0cca1-e525-4759-b8ad-bf2a82c5dec1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3d4a5be0-96a4-4cae-bb80-ec65cee94a22", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "bb1f2b699b61342ca13b75611df36eb9e3379ec4485edf0807dfbeb683246cf4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "57150321-dfaa-4fe3-a076-ae59685a4468", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "ae2231674f74efb1e249a6dc1f8e569d06c3c1fa35218e94b0d73dad2668c6c8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "17b258bf-820c-47fe-a6b9-1f60fdea0a48", "node_type": "1", "metadata": {}, "hash": "38ddc65233b05630bb4c2bb673684fdd350e4e603434d83be896bcdb52d438d9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 36723, "end_char_idx": 39430, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### 3.2 The key-value cache

Auto-regressive text generation with LLMs works by iteratively putting in an input sequence, sampling the next token, appending the next token to the input sequence, and continuing to do so until the LLM produces a token that signifies that the generation has finished.

Please have a look at [Transformer's Generate Text Tutorial](https://huggingface.co/docs/transformers/llm_tutorial#generate-text) to get a more visual explanation of how auto-regressive generation works.

Let's run a quick code snippet to show how auto-regressive works in practice. We will simply take the most likely next token via `torch.argmax`.

```python
input_ids = tokenizer(prompt, return_tensors="pt")["input_ids"].to("cuda")

for _ in range(5):
  next_logits = model(input_ids)["logits"][:, -1:]
  next_token_id = torch.argmax(next_logits,dim=-1)

  input_ids = torch.cat([input_ids, next_token_id], dim=-1)
  print("shape of input_ids", input_ids.shape)

generated_text = tokenizer.batch_decode(input_ids[:, -5:])
generated_text
```

**Output**:
```
shape of input_ids torch.Size([1, 21])
shape of input_ids torch.Size([1, 22])
shape of input_ids torch.Size([1, 23])
shape of input_ids torch.Size([1, 24])
shape of input_ids torch.Size([1, 25])
[' Here is a Python function']
```

As we can see every time we increase the text input tokens by the just sampled token.

With very few exceptions, LLMs are trained using the [causal language modeling objective](https://huggingface.co/docs/transformers/tasks/language_modeling#causal-language-modeling) and therefore mask the upper triangle matrix of the attention score - this is why in the two diagrams above the attention scores are left blank (*a.k.a* have 0 probability). For a quick recap on causal language modeling you can refer to the [*Illustrated Self Attention blog*](https://jalammar.github.io/illustrated-gpt2/#part-2-illustrated-self-attention).

As a consequence, tokens *never* depend on previous tokens, more specifically the \\( \mathbf{q}_i \\) vector is never put in relation with any key, values vectors \\( \mathbf{k}_j, \mathbf{v}_j \\) if \\( j > i \\) . Instead \\( \mathbf{q}_i \\) only attends to previous key-value vectors \\( \mathbf{k}_{m < i}, \mathbf{v}_{m < i} \text{ , for } m \in \{0, \ldots i - 1\} \\). In order to reduce unnecessary computation, one can therefore cache each layer's key-value vectors for all previous timesteps.

In the following, we will tell the LLM to make use of the key-value cache by retrieving and forwarding it for each forward pass.
In Transformers, we can retrieve the key-value cache by passing the `use_cache` flag to the `forward` call and can then pass it with the current token.


key: _node_content
string_value:
{"id_": "17b258bf-820c-47fe-a6b9-1f60fdea0a48", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3d4a5be0-96a4-4cae-bb80-ec65cee94a22", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "bb1f2b699b61342ca13b75611df36eb9e3379ec4485edf0807dfbeb683246cf4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d4e0cca1-e525-4759-b8ad-bf2a82c5dec1", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial_optimization", "title": "Optimizing LLMs for Speed and Memory", "tokens": 12767, "retrieve_doc": false, "source": "transformers"}, "hash": "e7070317de8c42cd1197a1f6ca5960c44f07fcf98e46ecc808a3ab01d0023fdf", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4e892dc1-6f80-440e-b629-d1b982778b5e", "node_type": "1", "metadata": {}, "hash": "c9cb2e02b42fc7b6d9c54967d96a5fedc624321fcba4b2a0494b55668d4ef2a9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 38103, "end_char_idx": 41062, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
With very few exceptions, LLMs are trained using the [causal language modeling objective](https://huggingface.co/docs/transformers/tasks/language_modeling#causal-language-modeling) and therefore mask the upper triangle matrix of the attention score - this is why in the two diagrams above the attention scores are left blank (*a.k.a* have 0 probability). For a quick recap on causal language modeling you can refer to the [*Illustrated Self Attention blog*](https://jalammar.github.io/illustrated-gpt2/#part-2-illustrated-self-attention).

As a consequence, tokens *never* depend on previous tokens, more specifically the \\( \mathbf{q}_i \\) vector is never put in relation with any key, values vectors \\( \mathbf{k}_j, \mathbf{v}_j \\) if \\( j > i \\) . Instead \\( \mathbf{q}_i \\) only attends to previous key-value vectors \\( \mathbf{k}_{m < i}, \mathbf{v}_{m < i} \text{ , for } m \in \{0, \ldots i - 1\} \\). In order to reduce unnecessary computation, one can therefore cache each layer's key-value vectors for all previous timesteps.

In the following, we will tell the LLM to make use of the key-value cache by retrieving and forwarding it for each forward pass.
In Transformers, we can retrieve the key-value cache by passing the `use_cache` flag to the `forward` call and can then pass it with the current token.

```python
past_key_values = None # past_key_values is the key-value cache
generated_tokens = []
next_token_id = tokenizer(prompt, return_tensors="pt")["input_ids"].to("cuda")

for _ in range(5):
  next_logits, past_key_values = model(next_token_id, past_key_values=past_key_values, use_cache=True).to_tuple()
  next_logits = next_logits[:, -1:]
  next_token_id = torch.argmax(next_logits, dim=-1)

  print("shape of input_ids", next_token_id.shape)
  print("length of key-value cache", len(past_key_values[0][0]))  # past_key_values are of shape [num_layers, 0 for k, 1 for v, batch_size, length, hidden_dim]
  generated_tokens.append(next_token_id.item())

generated_text = tokenizer.batch_decode(generated_tokens)
generated_text
```

**Output**:
```
shape of input_ids torch.Size([1, 1])
length of key-value cache 20
shape of input_ids torch.Size([1, 1])
length of key-value cache 21
shape of input_ids torch.Size([1, 1])
length of key-value cache 22
shape of input_ids torch.Size([1, 1])
length of key-value cache 23
shape of input_ids torch.Size([1, 1])
length of key-value cache 24
[' Here', ' is', ' a', ' Python', ' function']
```

As one can see, when using the key-value cache the text input tokens are *not* increased in length, but remain a single input vector. The length of the key-value cache on the other hand is increased by one at every decoding step.

> Making use of the key-value cache means that the \\( \mathbf{QK}^T \\) is essentially reduced to \\( \mathbf{q}_c\mathbf{K}^T \\) with \\( \mathbf{q}_c \\) being the query projection of the currently passed input token which is *always* just a single vector.


