# embedding_metadata

key: chroma:document
string_value:
The difference is the *`.default`* part in the second to last segment. This part exists because PEFT generally allows the addition of multiple adapters at once (using an `nn.ModuleDict` or `nn.ParameterDict` to store them). For example, if you add another adapter called "other", the key for that adapter would be `base_model.model.encoder.layer.0.attention.self.query.lora_A.other.weight`.

When you call [`~PeftModel.save_pretrained`], the adapter name is stripped from the keys. The reason is that the adapter name is not an important part of the model architecture; it is just an arbitrary name. When loading the adapter, you could choose a totally different name, and the model would still work the same way. This is why the adapter name is not stored in the checkpoint file.

<Tip>

If you call `save_pretrained("some/path")` and the adapter name is not `"default"`, the adapter is stored in a sub-directory with the same name as the adapter. So if the name is "other", it would be stored inside of `some/path/other`.

</Tip>

In some circumstances, deciding which values to add to the checkpoint file can become a bit more complicated. For example, in PEFT, DoRA is implemented as a special case of LoRA. If you want to convert a DoRA model to PEFT, you should create a LoRA checkpoint with extra entries for DoRA. You can see this in the `__init__` of the previous `LoraLayer` code:

```python
self.lora_magnitude_vector: Optional[torch.nn.ParameterDict] = None  # for DoRA
```

This indicates that there is an optional extra parameter per layer for DoRA.

### adapter_config

All the other information needed to load a PEFT model is contained in the `adapter_config.json` file. Let's check this file for a LoRA model applied to BERT:

```json
{
  "alpha_pattern": {},
  "auto_mapping": {
    "base_model_class": "BertModel",
    "parent_library": "transformers.models.bert.modeling_bert"
  },
  "base_model_name_or_path": "bert-base-uncased",
  "bias": "none",
  "fan_in_fan_out": false,
  "inference_mode": true,
  "init_lora_weights": true,
  "layer_replication": null,
  "layers_pattern": null,
  "layers_to_transform": null,
  "loftq_config": {},
  "lora_alpha": 8,
  "lora_dropout": 0.0,
  "megatron_config": null,
  "megatron_core": "megatron.core",
  "modules_to_save": null,
  "peft_type": "LORA",
  "r": 8,
  "rank_pattern": {},
  "revision": null,
  "target_modules": [
    "query",
    "value"
  ],
  "task_type": null,
  "use_dora": false,
  "use_rslora": false
}
```

This contains a lot of entries, and at first glance, it could feel overwhelming to figure out all the right values to put in there. However, most of the entries are not necessary to load the model. This is either because they use the default values and don't need to be added or because they only affect the initialization of the LoRA weights, which is irrelevant when it comes to loading the model. If you find that you don't know what a specific parameter does, e.g., `"use_rslora",` don't add it, and you should be fine.


key: _node_content
string_value:
{"id_": "899a3ab9-a1b6-4606-ac16-200851902e8f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/peft/developer_guides/checkpoint", "title": "PEFT checkpoint format", "tokens": 3329, "retrieve_doc": true, "source": "peft"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b712ff60-7453-4987-8e37-11c5db5d9b62", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/peft/developer_guides/checkpoint", "title": "PEFT checkpoint format", "tokens": 3329, "retrieve_doc": true, "source": "peft"}, "hash": "b5a16eafb9f7797e65ffc2b56e056a1313df363eff570e9b50f1b7654f395a7f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1d2183df-905c-462c-87f6-89c801856cba", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/peft/developer_guides/checkpoint", "title": "PEFT checkpoint format", "tokens": 3329, "retrieve_doc": true, "source": "peft"}, "hash": "bca7f26c0ea2aebd9ef7a5a8478610b858ed6f15b13c0d56b151beffb7d7996c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6f9d94c9-6e8c-4c50-8e70-d3fd387de710", "node_type": "1", "metadata": {}, "hash": "e064b178a5b5a4148beb15ff363677516edd059f7719afc0e85281b819b2717c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8891, "end_char_idx": 11981, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Let's check this file for a LoRA model applied to BERT:

```json
{
  "alpha_pattern": {},
  "auto_mapping": {
    "base_model_class": "BertModel",
    "parent_library": "transformers.models.bert.modeling_bert"
  },
  "base_model_name_or_path": "bert-base-uncased",
  "bias": "none",
  "fan_in_fan_out": false,
  "inference_mode": true,
  "init_lora_weights": true,
  "layer_replication": null,
  "layers_pattern": null,
  "layers_to_transform": null,
  "loftq_config": {},
  "lora_alpha": 8,
  "lora_dropout": 0.0,
  "megatron_config": null,
  "megatron_core": "megatron.core",
  "modules_to_save": null,
  "peft_type": "LORA",
  "r": 8,
  "rank_pattern": {},
  "revision": null,
  "target_modules": [
    "query",
    "value"
  ],
  "task_type": null,
  "use_dora": false,
  "use_rslora": false
}
```

This contains a lot of entries, and at first glance, it could feel overwhelming to figure out all the right values to put in there. However, most of the entries are not necessary to load the model. This is either because they use the default values and don't need to be added or because they only affect the initialization of the LoRA weights, which is irrelevant when it comes to loading the model. If you find that you don't know what a specific parameter does, e.g., `"use_rslora",` don't add it, and you should be fine. Also note that as more options are added, this file will get more entries in the future, but it should be backward compatible.

At the minimum, you should include the following entries:

```json
{
  "target_modules": ["query", "value"],
  "peft_type": "LORA"
}
```

However, adding as many entries as possible, like the rank `r` or the `base_model_name_or_path` (if it's a Transformers model) is recommended. This information can help others understand the model better and share it more easily. To check which keys and values are expected, check out the [config.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora/config.py) file (as an example, this is the config file for LoRA) in the PEFT source code.

## Model storage

In some circumstances, you might want to store the whole PEFT model, including the base weights. This can be necessary if, for instance, the base model is not available to the users trying to load the PEFT model. You can merge the weights first or convert it into a Transformer model.

### Merge the weights

The most straightforward way to store the whole PEFT model is to merge the adapter weights into the base weights:

```python
merged_model = model.merge_and_unload()
merged_model.save_pretrained(...)
```

There are some disadvantages to this approach, though:

- Once [`~LoraModel.merge_and_unload`] is called, you get a basic model without any PEFT-specific functionality. This means you can't use any of the PEFT-specific methods anymore.
- You cannot unmerge the weights, load multiple adapters at once, disable the adapter, etc.
- Not all PEFT methods support merging weights.
- Some PEFT methods may generally allow merging, but not with specific settings (e.g. when using certain quantization techniques).


key: _node_content
string_value:
{"id_": "6f9d94c9-6e8c-4c50-8e70-d3fd387de710", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/peft/developer_guides/checkpoint", "title": "PEFT checkpoint format", "tokens": 3329, "retrieve_doc": true, "source": "peft"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b712ff60-7453-4987-8e37-11c5db5d9b62", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/peft/developer_guides/checkpoint", "title": "PEFT checkpoint format", "tokens": 3329, "retrieve_doc": true, "source": "peft"}, "hash": "b5a16eafb9f7797e65ffc2b56e056a1313df363eff570e9b50f1b7654f395a7f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "899a3ab9-a1b6-4606-ac16-200851902e8f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/peft/developer_guides/checkpoint", "title": "PEFT checkpoint format", "tokens": 3329, "retrieve_doc": true, "source": "peft"}, "hash": "347709a5226a6fd79b522ae68fea5b6b96e7cdd729a05e0f46e20a38b7778587", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10346, "end_char_idx": 13198, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
At the minimum, you should include the following entries:

```json
{
  "target_modules": ["query", "value"],
  "peft_type": "LORA"
}
```

However, adding as many entries as possible, like the rank `r` or the `base_model_name_or_path` (if it's a Transformers model) is recommended. This information can help others understand the model better and share it more easily. To check which keys and values are expected, check out the [config.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora/config.py) file (as an example, this is the config file for LoRA) in the PEFT source code.

## Model storage

In some circumstances, you might want to store the whole PEFT model, including the base weights. This can be necessary if, for instance, the base model is not available to the users trying to load the PEFT model. You can merge the weights first or convert it into a Transformer model.

### Merge the weights

The most straightforward way to store the whole PEFT model is to merge the adapter weights into the base weights:

```python
merged_model = model.merge_and_unload()
merged_model.save_pretrained(...)
```

There are some disadvantages to this approach, though:

- Once [`~LoraModel.merge_and_unload`] is called, you get a basic model without any PEFT-specific functionality. This means you can't use any of the PEFT-specific methods anymore.
- You cannot unmerge the weights, load multiple adapters at once, disable the adapter, etc.
- Not all PEFT methods support merging weights.
- Some PEFT methods may generally allow merging, but not with specific settings (e.g. when using certain quantization techniques).
- The whole model will be much larger than the PEFT model, as it will contain all the base weights as well.

But inference with a merged model should be a bit faster.

### Convert to a Transformers model

Another way to save the whole model, assuming the base model is a Transformers model, is to use this hacky approach to directly insert the PEFT weights into the base model and save it, which only works if you "trick" Transformers into believing the PEFT model is not a PEFT model. This only works with LoRA because other adapters are not implemented in Transformers.

```python
model = ...  # the PEFT model
...
# after you finish training the model, save it in a temporary location
model.save_pretrained(<temp_location>)
# now load this model directly into a transformers model, without the PEFT wrapper
# the PEFT weights are directly injected into the base model
model_loaded = AutoModel.from_pretrained(<temp_location>)
# now make the loaded model believe that it is _not_ a PEFT model
model_loaded._hf_peft_config_loaded = False
# now when we save it, it will save the whole model
model_loaded.save_pretrained(<final_location>)
# or upload to Hugging Face Hub
model_loaded.push_to_hub(<final_location>)
```


key: _node_content
string_value:
{"id_": "c5b75e39-c697-44ee-80ae-e5777777f92f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/peft/developer_guides/contributing", "title": "Contribute to PEFT", "tokens": 1440, "retrieve_doc": true, "source": "peft"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "08aabeb0-2446-4c52-81fd-acdcbcc827f4", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/peft/developer_guides/contributing", "title": "Contribute to PEFT", "tokens": 1440, "retrieve_doc": true, "source": "peft"}, "hash": "e2e224fcf3925fdbf7dd2f42ab395bae6749d9b56a9ab6b0d7360707a7fd39b0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f6754473-005d-4250-9f54-d3a703dafc1d", "node_type": "1", "metadata": {}, "hash": "d26228fe18e8883457ac0706139fd2caa9a4d96c8204dcffd07d4638e60152cc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3341, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Contribute to PEFT

We are happy to accept contributions to PEFT. If you plan to contribute, please read this to make the process as smooth as possible.

## Installation

For code contributions to PEFT, you should choose the ["source"](../install#source) installation method.

If you are new to creating a pull request, follow the [Creating a pull request](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request) guide by GitHub.

## Tests and code quality checks

Regardless of the contribution type (unless it’s only about the docs), you should run tests and code quality checks before creating a PR to ensure your contribution doesn’t break anything and follows the project standards.

We provide a Makefile to execute the necessary tests. Run the code below for the unit test:

```sh
make test
```

Run one of the following to either only check or check and fix code quality and style:

```sh
make quality  # just check
make style  # check and fix
```

You can also set up [`pre-commit`](https://pre-commit.com/) to run these fixes
automatically as Git commit hooks.

```bash
$ pip install pre-commit
$ pre-commit install
```

Running all the tests can take a couple of minutes, so during development it can be more efficient to only run tests specific to your change:

```sh
pytest tests/ -k <name-of-test>
```

This should finish much quicker and allow for faster iteration. However, you should still run the whole test suite before creating a PR because your change can inadvertently break tests that at first glance are unrelated.

If your change is specific to a hardware setting (e.g., it requires CUDA), take a look at [tests/test_gpu_examples.py](https://github.com/huggingface/peft/blob/1c1c7fdaa6e6abaa53939b865dee1eded82ad032/tests/test_gpu_examples.py) and [tests/test_common_gpu.py](https://github.com/huggingface/peft/blob/1c1c7fdaa6e6abaa53939b865dee1eded82ad032/tests/test_common_gpu.py) to see if it makes sense to add tests there. If your change could have an effect on saving and loading models, please run the tests with the `--regression` flag to trigger regression tests.

It can happen that while you’re working on your PR, the underlying code base changes due to other changes being merged. If that happens – especially when there is a merge conflict – please update your branch with the latest changes. This can be a merge or a rebase, and we'll squash and merge the PR once it’s ready.

## PR description

When opening a PR, please provide a nice description of the change you're proposing. If it relates to other issues or PRs, please reference them. Providing a good description not only helps the reviewers review your code better and faster, it can also be used later (as a basis) for the commit message which helps with long term maintenance of the project.

If your code makes some non-trivial changes, it may also be a good idea to add comments to the code to explain those changes. For example, if you had to iterate on your implementation multiple times because the most obvious way didn’t work, it’s a good indication that a code comment is needed.

## Bugfixes

Please give a description of the circumstances that led to the bug. If there is an existing issue, please link to it (e.g., “Resolves #12345”).


key: _node_content
string_value:
{"id_": "f6754473-005d-4250-9f54-d3a703dafc1d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/peft/developer_guides/contributing", "title": "Contribute to PEFT", "tokens": 1440, "retrieve_doc": true, "source": "peft"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "08aabeb0-2446-4c52-81fd-acdcbcc827f4", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/peft/developer_guides/contributing", "title": "Contribute to PEFT", "tokens": 1440, "retrieve_doc": true, "source": "peft"}, "hash": "e2e224fcf3925fdbf7dd2f42ab395bae6749d9b56a9ab6b0d7360707a7fd39b0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c5b75e39-c697-44ee-80ae-e5777777f92f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/peft/developer_guides/contributing", "title": "Contribute to PEFT", "tokens": 1440, "retrieve_doc": true, "source": "peft"}, "hash": "d9e6e1fdc04861704e0781991b997df2d1b25014f9aecd7d61f2a819dd1631aa", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2660b15b-26ac-42a1-b728-9fea96bab82f", "node_type": "1", "metadata": {}, "hash": "58ed11205cbb8bb5c59ca172a99a3bd18e6df12364e057751a8faa6edac6bcca", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2048, "end_char_idx": 5581, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
If your change could have an effect on saving and loading models, please run the tests with the `--regression` flag to trigger regression tests.

It can happen that while you’re working on your PR, the underlying code base changes due to other changes being merged. If that happens – especially when there is a merge conflict – please update your branch with the latest changes. This can be a merge or a rebase, and we'll squash and merge the PR once it’s ready.

## PR description

When opening a PR, please provide a nice description of the change you're proposing. If it relates to other issues or PRs, please reference them. Providing a good description not only helps the reviewers review your code better and faster, it can also be used later (as a basis) for the commit message which helps with long term maintenance of the project.

If your code makes some non-trivial changes, it may also be a good idea to add comments to the code to explain those changes. For example, if you had to iterate on your implementation multiple times because the most obvious way didn’t work, it’s a good indication that a code comment is needed.

## Bugfixes

Please give a description of the circumstances that led to the bug. If there is an existing issue, please link to it (e.g., “Resolves #12345”).

Ideally when a bugfix is provided, it should be accompanied by a test for the bug. The test should fail with the current code and pass with the bugfix. Add a comment to the test that references the issue or PR. Without a test, it is more difficult to prevent regressions in the future.

## Add a new fine-tuning method

New parameter-efficient fine-tuning methods are developed all the time. If you would like to add a new and promising method to PEFT, please follow these steps.

1. Before you start to implement the new method, please open a GitHub issue with your proposal. This way, the maintainers can give you some early feedback.
2. Please add a link to the source (usually a paper) of the method. Some evidence should be provided there is general interest in using the method. We will not add new methods that are freshly published, but there is no evidence of demand for it.
3. When implementing the method, it makes sense to look for existing implementations that already exist as a guide. Moreover, when you structure your code, please take inspiration from the other PEFT methods. For example, if your method is similar to LoRA, it makes sense to structure your code similarly or even reuse some functions or classes where it makes sense (some code duplication is okay, but don’t overdo it).
4. Ideally, in addition to the implementation of the new method, there should also be examples (notebooks, scripts), documentation, and an extensive test suite that proves the method works with a variety of tasks. However, this can be more challenging so it is acceptable to only provide the implementation and at least one working example. Documentation and tests can be added in follow up PRs.
5. Once you have something that seems to be working, don’t hesitate to create a draft PR even if it’s not in a mergeable state yet. The maintainers are happy to give you feedback and guidance along the way.

## Add other features

It is best if you first open an issue on GitHub with a proposal to add the new feature. This way, you can discuss with the maintainers if it makes sense to add the feature before spending too much time on implementing it.

New features should generally be accompanied by tests and documentation or examples.


key: _node_content
string_value:
{"id_": "2660b15b-26ac-42a1-b728-9fea96bab82f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/peft/developer_guides/contributing", "title": "Contribute to PEFT", "tokens": 1440, "retrieve_doc": true, "source": "peft"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "08aabeb0-2446-4c52-81fd-acdcbcc827f4", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/peft/developer_guides/contributing", "title": "Contribute to PEFT", "tokens": 1440, "retrieve_doc": true, "source": "peft"}, "hash": "e2e224fcf3925fdbf7dd2f42ab395bae6749d9b56a9ab6b0d7360707a7fd39b0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f6754473-005d-4250-9f54-d3a703dafc1d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/peft/developer_guides/contributing", "title": "Contribute to PEFT", "tokens": 1440, "retrieve_doc": true, "source": "peft"}, "hash": "c8b1c5614a36ac9081491ea0909a6096ecc0f23209a88963d22fd6f007791704", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3735, "end_char_idx": 5830, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
If you would like to add a new and promising method to PEFT, please follow these steps.

1. Before you start to implement the new method, please open a GitHub issue with your proposal. This way, the maintainers can give you some early feedback.
2. Please add a link to the source (usually a paper) of the method. Some evidence should be provided there is general interest in using the method. We will not add new methods that are freshly published, but there is no evidence of demand for it.
3. When implementing the method, it makes sense to look for existing implementations that already exist as a guide. Moreover, when you structure your code, please take inspiration from the other PEFT methods. For example, if your method is similar to LoRA, it makes sense to structure your code similarly or even reuse some functions or classes where it makes sense (some code duplication is okay, but don’t overdo it).
4. Ideally, in addition to the implementation of the new method, there should also be examples (notebooks, scripts), documentation, and an extensive test suite that proves the method works with a variety of tasks. However, this can be more challenging so it is acceptable to only provide the implementation and at least one working example. Documentation and tests can be added in follow up PRs.
5. Once you have something that seems to be working, don’t hesitate to create a draft PR even if it’s not in a mergeable state yet. The maintainers are happy to give you feedback and guidance along the way.

## Add other features

It is best if you first open an issue on GitHub with a proposal to add the new feature. This way, you can discuss with the maintainers if it makes sense to add the feature before spending too much time on implementing it.

New features should generally be accompanied by tests and documentation or examples. Without the latter, users will have a hard time discovering your cool new feature.

Changes to the code should be implemented in a backward-compatible way. For example, existing code should continue to work the same way after the feature is merged.


key: _node_content
string_value:
{"id_": "3c8aed03-d3d7-48d0-901b-3cc757e89b4f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/peft/developer_guides/low_level_api", "title": "Adapter injection", "tokens": 925, "retrieve_doc": true, "source": "peft"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7bdab009-add7-41b9-802e-0350c46ceb0e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/peft/developer_guides/low_level_api", "title": "Adapter injection", "tokens": 925, "retrieve_doc": true, "source": "peft"}, "hash": "8c5c5ea776aaac2fa439ef2f6e3c6acebef075ea832f2eb32be99492e4462c4d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3022, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Adapter injection

With PEFT, you can inject trainable adapters into any `torch` module which allows you to use adapter methods without relying on the modeling classes in PEFT. Currently, PEFT supports injecting [LoRA](../conceptual_guides/adapter#low-rank-adaptation-lora), [AdaLoRA](../conceptual_guides/adapter#adaptive-low-rank-adaptation-adalora), and [IA3](../conceptual_guides/ia3) into models because for these adapters, inplace modification of the model is sufficient for finetuning it.

Check the table below to see when you should inject adapters.

| Pros | Cons |
|---|---|
| the model is modified inplace, keeping all the original attributes and methods | manually write the `from_pretrained` and `save_pretrained` utility functions from Hugging Face to save and load adapters |
| works for any `torch` module and modality | doesn't work with any of the utility methods provided by `PeftModel` such as disabling and merging adapters |

To perform the adapter injection, use the [`inject_adapter_in_model`] method. This method takes 3 arguments, the PEFT config, the model, and an optional adapter name. You can also attach multiple adapters to the model if you call [`inject_adapter_in_model`] multiple times with different adapter names.

For example, to inject LoRA adapters into the `linear` submodule of the `DummyModel` module:

```python
import torch
from peft import inject_adapter_in_model, LoraConfig

class DummyModel(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.embedding = torch.nn.Embedding(10, 10)
        self.linear = torch.nn.Linear(10, 10)
        self.lm_head = torch.nn.Linear(10, 10)

    def forward(self, input_ids):
        x = self.embedding(input_ids)
        x = self.linear(x)
        x = self.lm_head(x)
        return x


lora_config = LoraConfig(
    lora_alpha=16,
    lora_dropout=0.1,
    r=64,
    bias="none",
    target_modules=["linear"],
)

model = DummyModel()
model = inject_adapter_in_model(lora_config, model)

dummy_inputs = torch.LongTensor([[0, 1, 2, 3, 4, 5, 6, 7]])
dummy_outputs = model(dummy_inputs)
```

Print the model to see that the adapters have been correctly injected.

```bash
DummyModel(
  (embedding): Embedding(10, 10)
  (linear): Linear(
    in_features=10, out_features=10, bias=True
    (lora_dropout): ModuleDict(
      (default): Dropout(p=0.1, inplace=False)
    )
    (lora_A): ModuleDict(
      (default): Linear(in_features=10, out_features=64, bias=False)
    )
    (lora_B): ModuleDict(
      (default): Linear(in_features=64, out_features=10, bias=False)
    )
    (lora_embedding_A): ParameterDict()
    (lora_embedding_B): ParameterDict()
  )
  (lm_head): Linear(in_features=10, out_features=10, bias=True)
)
```

To only save the adapter, use the [`get_peft_model_state_dict`] function:

```python
from peft import get_peft_model_state_dict

peft_state_dict = get_peft_model_state_dict(model)
print(peft_state_dict)
```

Otherwise, `model.state_dict()` returns the full state dict of the model.


key: _node_content
string_value:
{"id_": "4a6ff3dd-53d5-41ff-aee2-29338d464521", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/peft/developer_guides/quantization", "title": "Quantization", "tokens": 2386, "retrieve_doc": true, "source": "peft"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3ebb426b-1d4b-4fef-8587-26d4163598c8", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/peft/developer_guides/quantization", "title": "Quantization", "tokens": 2386, "retrieve_doc": true, "source": "peft"}, "hash": "a0e1d9a95950f5f13ba96b1183c0dd62ef6dfa413e31102af200dbb8aba4ac12", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "efbf3bcc-0dc8-41ac-8072-f53bdd2968ac", "node_type": "1", "metadata": {}, "hash": "1a9a7ebcb6ffe4028ccb09502d7f31923edbc186387036bc0a29c89d8cf66f2b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2924, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Quantization

Quantization represents data with fewer bits, making it a useful technique for reducing memory-usage and accelerating inference especially when it comes to large language models (LLMs). There are several ways to quantize a model including:

* optimizing which model weights are quantized with the [AWQ](https://hf.co/papers/2306.00978) algorithm
* independently quantizing each row of a weight matrix with the [GPTQ](https://hf.co/papers/2210.17323) algorithm
* quantizing to 8-bit and 4-bit precision with the [bitsandbytes](https://github.com/TimDettmers/bitsandbytes) library
* quantizing to as low as 2-bit precision with the [AQLM](https://arxiv.org/abs/2401.06118) algorithm

However, after a model is quantized it isn't typically further trained for downstream tasks because training can be unstable due to the lower precision of the weights and activations. But since PEFT methods only add *extra* trainable parameters, this allows you to train a quantized model with a PEFT adapter on top! Combining quantization with PEFT can be a good strategy for training even the largest models on a single GPU. For example, [QLoRA](https://hf.co/papers/2305.14314) is a method that quantizes a model to 4-bits and then trains it with LoRA. This method allows you to finetune a 65B parameter model on a single 48GB GPU!

In this guide, you'll see how to quantize a model to 4-bits and train it with LoRA.

## Quantize a model

[bitsandbytes](https://github.com/TimDettmers/bitsandbytes) is a quantization library with a Transformers integration. With this integration, you can quantize a model to 8 or 4-bits and enable many other options by configuring the [`~transformers.BitsAndBytesConfig`] class. For example, you can:

* set `load_in_4bit=True` to quantize the model to 4-bits when you load it
* set `bnb_4bit_quant_type="nf4"` to use a special 4-bit data type for weights initialized from a normal distribution
* set `bnb_4bit_use_double_quant=True` to use a nested quantization scheme to quantize the already quantized weights
* set `bnb_4bit_compute_dtype=torch.bfloat16` to use bfloat16 for faster computation

```py
import torch
from transformers import BitsAndBytesConfig

config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)
```

Pass the `config` to the [`~transformers.AutoModelForCausalLM.from_pretrained`] method.

```py
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-v0.1", quantization_config=config)
```

Next, you should call the [`~peft.utils.prepare_model_for_kbit_training`] function to preprocess the quantized model for training.

```py
from peft import prepare_model_for_kbit_training

model = prepare_model_for_kbit_training(model)
```

Now that the quantized model is ready, let's set up a configuration.


key: _node_content
string_value:
{"id_": "efbf3bcc-0dc8-41ac-8072-f53bdd2968ac", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/peft/developer_guides/quantization", "title": "Quantization", "tokens": 2386, "retrieve_doc": true, "source": "peft"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3ebb426b-1d4b-4fef-8587-26d4163598c8", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/peft/developer_guides/quantization", "title": "Quantization", "tokens": 2386, "retrieve_doc": true, "source": "peft"}, "hash": "a0e1d9a95950f5f13ba96b1183c0dd62ef6dfa413e31102af200dbb8aba4ac12", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4a6ff3dd-53d5-41ff-aee2-29338d464521", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/peft/developer_guides/quantization", "title": "Quantization", "tokens": 2386, "retrieve_doc": true, "source": "peft"}, "hash": "ec33be5eab2cc7f5878c2aa12870392fad255d4958066dfa1b79ad32fe72a733", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "978083e4-5796-476b-8282-0718c0328798", "node_type": "1", "metadata": {}, "hash": "c226d5576c259d021250bd3ac3c5d186472d04749a0328145780f5f67627e376", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1419, "end_char_idx": 4373, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Quantize a model

[bitsandbytes](https://github.com/TimDettmers/bitsandbytes) is a quantization library with a Transformers integration. With this integration, you can quantize a model to 8 or 4-bits and enable many other options by configuring the [`~transformers.BitsAndBytesConfig`] class. For example, you can:

* set `load_in_4bit=True` to quantize the model to 4-bits when you load it
* set `bnb_4bit_quant_type="nf4"` to use a special 4-bit data type for weights initialized from a normal distribution
* set `bnb_4bit_use_double_quant=True` to use a nested quantization scheme to quantize the already quantized weights
* set `bnb_4bit_compute_dtype=torch.bfloat16` to use bfloat16 for faster computation

```py
import torch
from transformers import BitsAndBytesConfig

config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)
```

Pass the `config` to the [`~transformers.AutoModelForCausalLM.from_pretrained`] method.

```py
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-v0.1", quantization_config=config)
```

Next, you should call the [`~peft.utils.prepare_model_for_kbit_training`] function to preprocess the quantized model for training.

```py
from peft import prepare_model_for_kbit_training

model = prepare_model_for_kbit_training(model)
```

Now that the quantized model is ready, let's set up a configuration.

## LoraConfig

Create a [`LoraConfig`] with the following parameters (or choose your own):

```py
from peft import LoraConfig

config = LoraConfig(
    r=16,
    lora_alpha=8,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)
```

Then use the [`get_peft_model`] function to create a [`PeftModel`] from the quantized model and configuration.

```py
from peft import get_peft_model

model = get_peft_model(model, config)
```

You're all set for training with whichever training method you prefer!

### LoftQ initialization

[LoftQ](https://hf.co/papers/2310.08659) initializes LoRA weights such that the quantization error is minimized, and it can improve performance when training quantized models. To get started, follow [these instructions](https://github.com/huggingface/peft/tree/main/examples/loftq_finetuning).

In general, for LoftQ to work best, it is recommended to target as many layers with LoRA as possible, since those not targeted cannot have LoftQ applied. This means that passing `LoraConfig(..., target_modules="all-linear")` will most likely give the best results. Also, you should use `nf4` as quant type in your quantization config when using 4bit quantization, i.e. `BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4")`.

### QLoRA-style training

QLoRA adds trainable weights to all the linear layers in the transformer architecture.


key: _node_content
string_value:
{"id_": "978083e4-5796-476b-8282-0718c0328798", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/peft/developer_guides/quantization", "title": "Quantization", "tokens": 2386, "retrieve_doc": true, "source": "peft"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3ebb426b-1d4b-4fef-8587-26d4163598c8", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/peft/developer_guides/quantization", "title": "Quantization", "tokens": 2386, "retrieve_doc": true, "source": "peft"}, "hash": "a0e1d9a95950f5f13ba96b1183c0dd62ef6dfa413e31102af200dbb8aba4ac12", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "efbf3bcc-0dc8-41ac-8072-f53bdd2968ac", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/peft/developer_guides/quantization", "title": "Quantization", "tokens": 2386, "retrieve_doc": true, "source": "peft"}, "hash": "168d62c01a3b8b7c91446eee24b4fafd30475c11230aa8c01b202b79acd03018", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7a624d87-d09e-40d1-981d-777bf8a5099a", "node_type": "1", "metadata": {}, "hash": "99037bae5f897518b172e4165a96a7c070c05c2017d8dcfc5c456869ec298aca", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2926, "end_char_idx": 5505, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## LoraConfig

Create a [`LoraConfig`] with the following parameters (or choose your own):

```py
from peft import LoraConfig

config = LoraConfig(
    r=16,
    lora_alpha=8,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)
```

Then use the [`get_peft_model`] function to create a [`PeftModel`] from the quantized model and configuration.

```py
from peft import get_peft_model

model = get_peft_model(model, config)
```

You're all set for training with whichever training method you prefer!

### LoftQ initialization

[LoftQ](https://hf.co/papers/2310.08659) initializes LoRA weights such that the quantization error is minimized, and it can improve performance when training quantized models. To get started, follow [these instructions](https://github.com/huggingface/peft/tree/main/examples/loftq_finetuning).

In general, for LoftQ to work best, it is recommended to target as many layers with LoRA as possible, since those not targeted cannot have LoftQ applied. This means that passing `LoraConfig(..., target_modules="all-linear")` will most likely give the best results. Also, you should use `nf4` as quant type in your quantization config when using 4bit quantization, i.e. `BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4")`.

### QLoRA-style training

QLoRA adds trainable weights to all the linear layers in the transformer architecture. Since the attribute names for these linear layers can vary across architectures, set `target_modules` to `"all-linear"` to add LoRA to all the linear layers:

```py
config = LoraConfig(target_modules="all-linear", ...)
```

## AQLM quantization

Additive Quantization of Language Models ([AQLM](https://arxiv.org/abs/2401.06118)) is a Large Language Models compression method. It quantizes multiple weights together and takes advantage of interdependencies between them. AQLM represents groups of 8-16 weights as a sum of multiple vector codes. This allows it to compress models down to as low as 2-bit with considerably low accuracy losses.

Since the AQLM quantization process is computationally expensive, a use of prequantized models is recommended. A partial list of available models can be found in the official aqlm [repository](https://github.com/Vahe1994/AQLM).

The models support LoRA adapter tuning. To tune the quantized model you'll need to install the `aqlm` inference library: `pip install aqlm>=1.0.2`. Finetuned LoRA adapters shall be saved separately, as merging them with AQLM quantized weights is not possible.


key: _node_content
string_value:
{"id_": "7a624d87-d09e-40d1-981d-777bf8a5099a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/peft/developer_guides/quantization", "title": "Quantization", "tokens": 2386, "retrieve_doc": true, "source": "peft"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3ebb426b-1d4b-4fef-8587-26d4163598c8", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/peft/developer_guides/quantization", "title": "Quantization", "tokens": 2386, "retrieve_doc": true, "source": "peft"}, "hash": "a0e1d9a95950f5f13ba96b1183c0dd62ef6dfa413e31102af200dbb8aba4ac12", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "978083e4-5796-476b-8282-0718c0328798", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/peft/developer_guides/quantization", "title": "Quantization", "tokens": 2386, "retrieve_doc": true, "source": "peft"}, "hash": "2f878234432df6d23f5a8818ccd22cbd6f2bb0f6a2535198bb5fa7724a27e7ea", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1f163380-eda1-436b-a238-14a3a14b7a99", "node_type": "1", "metadata": {}, "hash": "f71e3ebe0fddc6ee90966bafd70b6b89eaef715697f78ba388f96dc9ddc21982", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3977, "end_char_idx": 6558, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This means that passing `LoraConfig(..., target_modules="all-linear")` will most likely give the best results. Also, you should use `nf4` as quant type in your quantization config when using 4bit quantization, i.e. `BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4")`.

### QLoRA-style training

QLoRA adds trainable weights to all the linear layers in the transformer architecture. Since the attribute names for these linear layers can vary across architectures, set `target_modules` to `"all-linear"` to add LoRA to all the linear layers:

```py
config = LoraConfig(target_modules="all-linear", ...)
```

## AQLM quantization

Additive Quantization of Language Models ([AQLM](https://arxiv.org/abs/2401.06118)) is a Large Language Models compression method. It quantizes multiple weights together and takes advantage of interdependencies between them. AQLM represents groups of 8-16 weights as a sum of multiple vector codes. This allows it to compress models down to as low as 2-bit with considerably low accuracy losses.

Since the AQLM quantization process is computationally expensive, a use of prequantized models is recommended. A partial list of available models can be found in the official aqlm [repository](https://github.com/Vahe1994/AQLM).

The models support LoRA adapter tuning. To tune the quantized model you'll need to install the `aqlm` inference library: `pip install aqlm>=1.0.2`. Finetuned LoRA adapters shall be saved separately, as merging them with AQLM quantized weights is not possible.

```py
quantized_model = AutoModelForCausalLM.from_pretrained(
    "BlackSamorez/Mixtral-8x7b-AQLM-2Bit-1x16-hf-test-dispatch",
    torch_dtype="auto", device_map="auto", low_cpu_mem_usage=True,
)

peft_config = LoraConfig(...)

quantized_model = get_peft_model(quantized_model, peft_config)
```

You can refer to the [Google Colab](https://colab.research.google.com/drive/12GTp1FCj5_0SnnNQH18h_2XFh9vS_guX?usp=sharing) example for an overview of AQLM+LoRA finetuning.

## EETQ quantization

You can also perform LoRA fine-tuning on EETQ quantized models. [EETQ](https://github.com/NetEase-FuXi/EETQ) package offers simple and efficient way to perform 8-bit quantization, which is claimed to be faster than the `LLM.int8()` algorithm. First, make sure that you have a transformers version that is compatible with EETQ (e.g. by installing it from latest pypi or from source).

```py
import torch
from transformers import EetqConfig

config = EetqConfig("int8")
```

Pass the `config` to the [`~transformers.AutoModelForCausalLM.from_pretrained`] method.


key: _node_content
string_value:
{"id_": "1f163380-eda1-436b-a238-14a3a14b7a99", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/peft/developer_guides/quantization", "title": "Quantization", "tokens": 2386, "retrieve_doc": true, "source": "peft"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3ebb426b-1d4b-4fef-8587-26d4163598c8", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/peft/developer_guides/quantization", "title": "Quantization", "tokens": 2386, "retrieve_doc": true, "source": "peft"}, "hash": "a0e1d9a95950f5f13ba96b1183c0dd62ef6dfa413e31102af200dbb8aba4ac12", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7a624d87-d09e-40d1-981d-777bf8a5099a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/peft/developer_guides/quantization", "title": "Quantization", "tokens": 2386, "retrieve_doc": true, "source": "peft"}, "hash": "1cc39db77d90bf1b61af264e1bc2f67e5c2c958f9c69680cf8d94018caeb1c8d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e69f1907-5094-40cb-ae52-a83f38bf40d6", "node_type": "1", "metadata": {}, "hash": "f7b8ee426f627e160d53834ff96e8b8b0f9b34f0172305527ce338d742346406", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5246, "end_char_idx": 7695, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The models support LoRA adapter tuning. To tune the quantized model you'll need to install the `aqlm` inference library: `pip install aqlm>=1.0.2`. Finetuned LoRA adapters shall be saved separately, as merging them with AQLM quantized weights is not possible.

```py
quantized_model = AutoModelForCausalLM.from_pretrained(
    "BlackSamorez/Mixtral-8x7b-AQLM-2Bit-1x16-hf-test-dispatch",
    torch_dtype="auto", device_map="auto", low_cpu_mem_usage=True,
)

peft_config = LoraConfig(...)

quantized_model = get_peft_model(quantized_model, peft_config)
```

You can refer to the [Google Colab](https://colab.research.google.com/drive/12GTp1FCj5_0SnnNQH18h_2XFh9vS_guX?usp=sharing) example for an overview of AQLM+LoRA finetuning.

## EETQ quantization

You can also perform LoRA fine-tuning on EETQ quantized models. [EETQ](https://github.com/NetEase-FuXi/EETQ) package offers simple and efficient way to perform 8-bit quantization, which is claimed to be faster than the `LLM.int8()` algorithm. First, make sure that you have a transformers version that is compatible with EETQ (e.g. by installing it from latest pypi or from source).

```py
import torch
from transformers import EetqConfig

config = EetqConfig("int8")
```

Pass the `config` to the [`~transformers.AutoModelForCausalLM.from_pretrained`] method.

```py
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-v0.1", quantization_config=config)
```

and create a `LoraConfig` and pass it to `get_peft_model`:

```py
from peft import LoraConfig, get_peft_model

config = LoraConfig(
    r=16,
    lora_alpha=8,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

model = get_peft_model(model, config)
```

## HQQ quantization

The models that is quantized using Half-Quadratic Quantization of Large Machine Learning Models ([HQQ](https://mobiusml.github.io/hqq_blog/)) support LoRA adapter tuning. To tune the quantized model, you'll need to install the `hqq` library with: `pip install hqq`.

```py
from hqq.engine.hf import HQQModelForCausalLM

quantized_model = HQQModelForCausalLM.from_quantized(save_dir_or_hfhub, device='cuda')

peft_config = LoraConfig(...)

quantized_model = get_peft_model(quantized_model, peft_config)
```

Or using transformers version that is compatible with HQQ (e.g. by installing it from latest pypi or from source).


key: _node_content
string_value:
{"id_": "e69f1907-5094-40cb-ae52-a83f38bf40d6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/peft/developer_guides/quantization", "title": "Quantization", "tokens": 2386, "retrieve_doc": true, "source": "peft"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3ebb426b-1d4b-4fef-8587-26d4163598c8", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/peft/developer_guides/quantization", "title": "Quantization", "tokens": 2386, "retrieve_doc": true, "source": "peft"}, "hash": "a0e1d9a95950f5f13ba96b1183c0dd62ef6dfa413e31102af200dbb8aba4ac12", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1f163380-eda1-436b-a238-14a3a14b7a99", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/peft/developer_guides/quantization", "title": "Quantization", "tokens": 2386, "retrieve_doc": true, "source": "peft"}, "hash": "0f9c4366e57d70e5ae2d629a211354b186d0e1576af155c3268db90ef7dcc93d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6330, "end_char_idx": 8527, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
by installing it from latest pypi or from source).

```py
import torch
from transformers import EetqConfig

config = EetqConfig("int8")
```

Pass the `config` to the [`~transformers.AutoModelForCausalLM.from_pretrained`] method.

```py
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-v0.1", quantization_config=config)
```

and create a `LoraConfig` and pass it to `get_peft_model`:

```py
from peft import LoraConfig, get_peft_model

config = LoraConfig(
    r=16,
    lora_alpha=8,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

model = get_peft_model(model, config)
```

## HQQ quantization

The models that is quantized using Half-Quadratic Quantization of Large Machine Learning Models ([HQQ](https://mobiusml.github.io/hqq_blog/)) support LoRA adapter tuning. To tune the quantized model, you'll need to install the `hqq` library with: `pip install hqq`.

```py
from hqq.engine.hf import HQQModelForCausalLM

quantized_model = HQQModelForCausalLM.from_quantized(save_dir_or_hfhub, device='cuda')

peft_config = LoraConfig(...)

quantized_model = get_peft_model(quantized_model, peft_config)
```

Or using transformers version that is compatible with HQQ (e.g. by installing it from latest pypi or from source).

```python
from transformers import HqqConfig, AutoModelForCausalLM

quant_config = HqqConfig(nbits=4, group_size=64)

quantized_model = AutoModelForCausalLM.from_pretrained(save_dir_or_hfhub, device='cuda', quantization_config=quant_config)

peft_config = LoraConfig(...)

quantized_model = get_peft_model(quantized_model, peft_config)
```

## Next steps

If you're interested in learning more about quantization, the following may be helpful:

* Learn more about details about QLoRA and check out some benchmarks on its impact in the [Making LLMs even more accessible with bitsandbytes, 4-bit quantization and QLoRA](https://huggingface.co/blog/4bit-transformers-bitsandbytes) blog post.
* Read more about different quantization schemes in the Transformers [Quantization](https://hf.co/docs/transformers/main/quantization) guide.


key: _node_content
string_value:
{"id_": "15ec82ec-0141-467e-bbb1-a1c4755cccac", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/peft/developer_guides/model_merging", "title": "Model merging", "tokens": 2033, "retrieve_doc": true, "source": "peft"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6b02cbc4-d23f-42db-b98f-604f435db4d2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/peft/developer_guides/model_merging", "title": "Model merging", "tokens": 2033, "retrieve_doc": true, "source": "peft"}, "hash": "037d7dabf7d6c167c55600c9cf6f9be0e655a76409c0718de353a56c34df052b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f4e8145c-cbe4-4cb9-a0ca-abfc4a7cdf30", "node_type": "1", "metadata": {}, "hash": "3bcebc537ff6e9be53d47053aa2124f98f7bdc230ce4d9af39602fbf09d10cce", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2909, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Model merging

Training a model for each task can be costly, take up storage space, and the models aren't able to learn new information to improve their performance. Multitask learning can overcome some of these limitations by training a model to learn several tasks, but it is expensive to train and designing a dataset for it is challenging. *Model merging* offers a solution to these challenges by combining multiple pretrained models into one model, giving it the combined abilities of each individual model without any additional training.

PEFT provides several methods for merging models like a linear or SVD combination. This guide focuses on two methods that are more efficient for merging LoRA adapters by eliminating redundant parameters:

* [TIES](https://hf.co/papers/2306.01708) - TrIm, Elect, and Merge (TIES) is a three-step method for merging models. First, redundant parameters are trimmed, then conflicting signs are resolved into an aggregated vector, and finally the parameters whose signs are the same as the aggregate sign are averaged. This method takes into account that some values (redundant and sign disagreement) can degrade performance in the merged model.
* [DARE](https://hf.co/papers/2311.03099) - Drop And REscale is a method that can be used to prepare for other model merging methods like TIES. It works by randomly dropping parameters according to a drop rate and rescaling the remaining parameters. This helps to reduce the number of redundant and potentially interfering parameters among multiple models.

Models are merged with the [`~LoraModel.add_weighted_adapter`] method, and the specific model merging method is specified in the `combination_type` parameter.

## Merge method

With TIES and DARE, merging is enabled by setting `combination_type` and `density` to a value of the weights to keep from the individual models. For example, let's merge three finetuned [TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T](https://huggingface.co/TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T) models: [tinyllama_lora_nobots](https://huggingface.co/smangrul/tinyllama_lora_norobots), [tinyllama_lora_sql](https://huggingface.co/smangrul/tinyllama_lora_sql), and [tinyllama_lora_adcopy](https://huggingface.co/smangrul/tinyllama_lora_adcopy).

<Tip warninig={true}>

When you're attempting to merge fully trained models with TIES, you should be aware of any special tokens each model may have added to the embedding layer which are not a part of the original checkpoint's vocabulary. This may cause an issue because each model may have added a special token to the same embedding position. If this is the case, you should use the [`~transformers.PreTrainedModel.resize_token_embeddings`] method to avoid merging the special tokens at the same embedding index.

<br>

This shouldn't be an issue if you're only merging LoRA adapters trained from the same base model.


key: _node_content
string_value:
{"id_": "f4e8145c-cbe4-4cb9-a0ca-abfc4a7cdf30", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/peft/developer_guides/model_merging", "title": "Model merging", "tokens": 2033, "retrieve_doc": true, "source": "peft"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6b02cbc4-d23f-42db-b98f-604f435db4d2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/peft/developer_guides/model_merging", "title": "Model merging", "tokens": 2033, "retrieve_doc": true, "source": "peft"}, "hash": "037d7dabf7d6c167c55600c9cf6f9be0e655a76409c0718de353a56c34df052b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "15ec82ec-0141-467e-bbb1-a1c4755cccac", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/peft/developer_guides/model_merging", "title": "Model merging", "tokens": 2033, "retrieve_doc": true, "source": "peft"}, "hash": "5341ce8a43d6c3dcc4de1447cf247a2d6c75eb8ae063d841d7a3013b9f565bb0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f8a7f1e4-4239-46f8-81e1-01d92686d68a", "node_type": "1", "metadata": {}, "hash": "ff1be3560bc7817c182d63de6ddd7a314785a4cec6f84285b9d0ead527dd0fac", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1333, "end_char_idx": 4049, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
It works by randomly dropping parameters according to a drop rate and rescaling the remaining parameters. This helps to reduce the number of redundant and potentially interfering parameters among multiple models.

Models are merged with the [`~LoraModel.add_weighted_adapter`] method, and the specific model merging method is specified in the `combination_type` parameter.

## Merge method

With TIES and DARE, merging is enabled by setting `combination_type` and `density` to a value of the weights to keep from the individual models. For example, let's merge three finetuned [TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T](https://huggingface.co/TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T) models: [tinyllama_lora_nobots](https://huggingface.co/smangrul/tinyllama_lora_norobots), [tinyllama_lora_sql](https://huggingface.co/smangrul/tinyllama_lora_sql), and [tinyllama_lora_adcopy](https://huggingface.co/smangrul/tinyllama_lora_adcopy).

<Tip warninig={true}>

When you're attempting to merge fully trained models with TIES, you should be aware of any special tokens each model may have added to the embedding layer which are not a part of the original checkpoint's vocabulary. This may cause an issue because each model may have added a special token to the same embedding position. If this is the case, you should use the [`~transformers.PreTrainedModel.resize_token_embeddings`] method to avoid merging the special tokens at the same embedding index.

<br>

This shouldn't be an issue if you're only merging LoRA adapters trained from the same base model.

</Tip>

Load a base model and can use the [`~PeftModel.load_adapter`] method to load and assign each adapter a name:

```py
from peft import PeftConfig, PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

config = PeftConfig.from_pretrained("smangrul/tinyllama_lora_norobots")
model = AutoModelForCausalLM.from_pretrained(config.base_model_name_or_path, load_in_4bit=True, device_map="auto").eval()
tokenizer = AutoTokenizer.from_pretrained("smangrul/tinyllama_lora_norobots")

model = PeftModel.from_pretrained(model, "smangrul/tinyllama_lora_norobots", adapter_name="norobots")
_ = model.load_adapter("smangrul/tinyllama_lora_sql", adapter_name="sql")
_ = model.load_adapter("smangrul/tinyllama_lora_adcopy", adapter_name="adcopy")
```

Set the adapters, weights, `adapter_name`, `combination_type`, and `density` with the [`~LoraModel.add_weighted_adapter`] method.

<hfoptions id="merge-method">
<hfoption id="TIES">

Weight values greater than `1.0` typically produce better results because they preserve the correct scale. A good default starting value for the weights is to set all values to `1.0`.


key: _node_content
string_value:
{"id_": "f8a7f1e4-4239-46f8-81e1-01d92686d68a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/peft/developer_guides/model_merging", "title": "Model merging", "tokens": 2033, "retrieve_doc": true, "source": "peft"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6b02cbc4-d23f-42db-b98f-604f435db4d2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/peft/developer_guides/model_merging", "title": "Model merging", "tokens": 2033, "retrieve_doc": true, "source": "peft"}, "hash": "037d7dabf7d6c167c55600c9cf6f9be0e655a76409c0718de353a56c34df052b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f4e8145c-cbe4-4cb9-a0ca-abfc4a7cdf30", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/peft/developer_guides/model_merging", "title": "Model merging", "tokens": 2033, "retrieve_doc": true, "source": "peft"}, "hash": "4d37961ad26c7c14eba2c095bba16d00437480c051178e7aec8d70e3da076cf2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "68634f92-133d-4636-86dd-e8fb31f3f578", "node_type": "1", "metadata": {}, "hash": "43a515ea3df3b8b92306d651d7a09e5d53c1ef792a4347552b44712293104117", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2529, "end_char_idx": 4912, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This may cause an issue because each model may have added a special token to the same embedding position. If this is the case, you should use the [`~transformers.PreTrainedModel.resize_token_embeddings`] method to avoid merging the special tokens at the same embedding index.

<br>

This shouldn't be an issue if you're only merging LoRA adapters trained from the same base model.

</Tip>

Load a base model and can use the [`~PeftModel.load_adapter`] method to load and assign each adapter a name:

```py
from peft import PeftConfig, PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

config = PeftConfig.from_pretrained("smangrul/tinyllama_lora_norobots")
model = AutoModelForCausalLM.from_pretrained(config.base_model_name_or_path, load_in_4bit=True, device_map="auto").eval()
tokenizer = AutoTokenizer.from_pretrained("smangrul/tinyllama_lora_norobots")

model = PeftModel.from_pretrained(model, "smangrul/tinyllama_lora_norobots", adapter_name="norobots")
_ = model.load_adapter("smangrul/tinyllama_lora_sql", adapter_name="sql")
_ = model.load_adapter("smangrul/tinyllama_lora_adcopy", adapter_name="adcopy")
```

Set the adapters, weights, `adapter_name`, `combination_type`, and `density` with the [`~LoraModel.add_weighted_adapter`] method.

<hfoptions id="merge-method">
<hfoption id="TIES">

Weight values greater than `1.0` typically produce better results because they preserve the correct scale. A good default starting value for the weights is to set all values to `1.0`.

```py
adapters = ["norobots", "adcopy", "sql"]
weights = [2.0, 1.0, 1.0]
adapter_name = "merge"
density = 0.2
model.add_weighted_adapter(adapters, weights, adapter_name, combination_type="ties", density=density)
```

</hfoption>
<hfoption id="DARE">

```py
adapters = ["norobots", "adcopy", "sql"]
weights = [2.0, 0.3, 0.7]
adapter_name = "merge"
density = 0.2
model.add_weighted_adapter(adapters, weights, adapter_name, combination_type="dare_ties", density=density)
```

</hfoption>
</hfoptions>

Set the newly merged model as the active model with the [`~LoraModel.set_adapter`] method.

```py
model.set_adapter("merge")
```

Now you can use the merged model as an instruction-tuned model to write ad copy or SQL queries!

<hfoptions id="ties">
<hfoption id="instruct">

```py
messages = [
    {"role": "user", "content": "Write an essay about Generative AI.


key: _node_content
string_value:
{"id_": "68634f92-133d-4636-86dd-e8fb31f3f578", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/peft/developer_guides/model_merging", "title": "Model merging", "tokens": 2033, "retrieve_doc": true, "source": "peft"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6b02cbc4-d23f-42db-b98f-604f435db4d2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/peft/developer_guides/model_merging", "title": "Model merging", "tokens": 2033, "retrieve_doc": true, "source": "peft"}, "hash": "037d7dabf7d6c167c55600c9cf6f9be0e655a76409c0718de353a56c34df052b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f8a7f1e4-4239-46f8-81e1-01d92686d68a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/peft/developer_guides/model_merging", "title": "Model merging", "tokens": 2033, "retrieve_doc": true, "source": "peft"}, "hash": "f9caf5b50362604a325b3cf8d63f816b5beaf35cbed9699d85fa062b98b19970", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "20fd410a-9c39-4060-bafe-e3581514d34c", "node_type": "1", "metadata": {}, "hash": "be6fd733379a609f5a96dc091a2fe36e56c82e29baa2269fbc952ed427bec66a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3814, "end_char_idx": 6559, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<hfoptions id="merge-method">
<hfoption id="TIES">

Weight values greater than `1.0` typically produce better results because they preserve the correct scale. A good default starting value for the weights is to set all values to `1.0`.

```py
adapters = ["norobots", "adcopy", "sql"]
weights = [2.0, 1.0, 1.0]
adapter_name = "merge"
density = 0.2
model.add_weighted_adapter(adapters, weights, adapter_name, combination_type="ties", density=density)
```

</hfoption>
<hfoption id="DARE">

```py
adapters = ["norobots", "adcopy", "sql"]
weights = [2.0, 0.3, 0.7]
adapter_name = "merge"
density = 0.2
model.add_weighted_adapter(adapters, weights, adapter_name, combination_type="dare_ties", density=density)
```

</hfoption>
</hfoptions>

Set the newly merged model as the active model with the [`~LoraModel.set_adapter`] method.

```py
model.set_adapter("merge")
```

Now you can use the merged model as an instruction-tuned model to write ad copy or SQL queries!

<hfoptions id="ties">
<hfoption id="instruct">

```py
messages = [
    {"role": "user", "content": "Write an essay about Generative AI."},
]
text = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
inputs = tokenizer(text, return_tensors="pt")
inputs = {k: v.to("cuda") for k, v in inputs.items()}
outputs = model.generate(**inputs, max_new_tokens=256, do_sample=True, top_p=0.95, temperature=0.2, repetition_penalty=1.2, eos_token_id=tokenizer.eos_token_id)
print(tokenizer.decode(outputs[0]))
```

</hfoption>
<hfoption id="ad copy">

```py
messages = [
    {"role": "system", "content": "Create a text ad given the following product and description."},
    {"role": "user", "content": "Product: Sony PS5 PlayStation Console\nDescription: The PS5 console unleashes new gaming possibilities that you never anticipated."},
]
text = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
inputs = tokenizer(text, return_tensors="pt")
inputs = {k: v.to("cuda") for k, v in inputs.items()}
outputs = model.generate(**inputs, max_new_tokens=128, do_sample=True, top_p=0.95, temperature=0.2, repetition_penalty=1.2, eos_token_id=tokenizer.eos_token_id)
print(tokenizer.decode(outputs[0]))
```

</hfoption>
<hfoption id="SQL">

```py
text = """Table: 2-11365528-2
Columns: ['Team', 'Head Coach', 'President', 'Home Ground', 'Location']
Natural Query: Who is the Head Coach of the team whose President is Mario Volarevic?
SQL Query:"""

inputs = tokenizer(text, return_tensors="pt")
inputs = {k: v.to("cuda") for k, v in inputs.items()}
outputs = model.generate(**inputs, max_new_tokens=64, repetition_penalty=1.1, eos_token_id=tokenizer("</s>").input_ids[-1])
print(tokenizer.decode(outputs[0]))
```

</hfoption>
</hfoptions>


key: _node_content
string_value:
{"id_": "20fd410a-9c39-4060-bafe-e3581514d34c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/peft/developer_guides/model_merging", "title": "Model merging", "tokens": 2033, "retrieve_doc": true, "source": "peft"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6b02cbc4-d23f-42db-b98f-604f435db4d2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/peft/developer_guides/model_merging", "title": "Model merging", "tokens": 2033, "retrieve_doc": true, "source": "peft"}, "hash": "037d7dabf7d6c167c55600c9cf6f9be0e655a76409c0718de353a56c34df052b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "68634f92-133d-4636-86dd-e8fb31f3f578", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/peft/developer_guides/model_merging", "title": "Model merging", "tokens": 2033, "retrieve_doc": true, "source": "peft"}, "hash": "5f7c6e2b3eb1cda91c84c30d9148f9e979d6e102453a2c458f864d5c4440df0a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5463, "end_char_idx": 7375, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
"},
    {"role": "user", "content": "Product: Sony PS5 PlayStation Console\nDescription: The PS5 console unleashes new gaming possibilities that you never anticipated."},
]
text = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
inputs = tokenizer(text, return_tensors="pt")
inputs = {k: v.to("cuda") for k, v in inputs.items()}
outputs = model.generate(**inputs, max_new_tokens=128, do_sample=True, top_p=0.95, temperature=0.2, repetition_penalty=1.2, eos_token_id=tokenizer.eos_token_id)
print(tokenizer.decode(outputs[0]))
```

</hfoption>
<hfoption id="SQL">

```py
text = """Table: 2-11365528-2
Columns: ['Team', 'Head Coach', 'President', 'Home Ground', 'Location']
Natural Query: Who is the Head Coach of the team whose President is Mario Volarevic?
SQL Query:"""

inputs = tokenizer(text, return_tensors="pt")
inputs = {k: v.to("cuda") for k, v in inputs.items()}
outputs = model.generate(**inputs, max_new_tokens=64, repetition_penalty=1.1, eos_token_id=tokenizer("</s>").input_ids[-1])
print(tokenizer.decode(outputs[0]))
```

</hfoption>
</hfoptions>


## Merging (IA)³ Models
The (IA)³ models facilitate linear merging of adapters. To merge adapters in an (IA)³ model, utilize the `add_weighted_adapter` method from the `IA3Model` class. This method is analogous to the `add_weighted_adapter` method used in `LoraModel`, with the key difference being the absence of the `combination_type` parameter. For example, to merge three (IA)³ adapters into a PEFT model, you would proceed as follows:

```py
adapters = ["adapter1", "adapter2", "adapter3"]
weights = [0.4, 0.3, 0.3]
adapter_name = "merge"
model.add_weighted_adapter(adapters, weights, adapter_name)
```

It is recommended that the weights sum to 1.0 to preserve the scale of the model. The merged model can then be set as the active model using the `set_adapter` method:

```py
model.set_adapter("merge")
```


key: _node_content
string_value:
{"id_": "0a603b7b-2be4-4a4b-991e-9c729bcce42a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/ia3", "title": "IA3", "tokens": 2278, "retrieve_doc": true, "source": "peft"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a6b0a0ad-22e1-4a79-b967-4815b61f9236", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/ia3", "title": "IA3", "tokens": 2278, "retrieve_doc": true, "source": "peft"}, "hash": "c1393fd481102f1fef41ff3d4b2f88fbdf1a56a876eaf96f6918e3274864df49", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "692a5108-4ae5-486b-a70b-acdcdfff7295", "node_type": "1", "metadata": {}, "hash": "c97a78c4dd7f0a68dc6810255cf39c7bdb5fb4b8277a973d492d3ae4b75ece5d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2447, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# IA3

[IA3](../conceptual_guides/ia3) multiplies the model's activations (the keys and values in the self-attention and encoder-decoder attention blocks, and the intermediate activation of the position-wise feedforward network) by three learned vectors. This PEFT method introduces an even smaller number of trainable parameters than LoRA which introduces weight matrices instead of vectors. The original model's parameters are kept frozen and only these vectors are updated. As a result, it is faster, cheaper and more efficient to finetune for a new downstream task.

This guide will show you how to train a sequence-to-sequence model with IA3 to *generate a sentiment* given some financial news.

<Tip>

Some familiarity with the general process of training a sequence-to-sequence would be really helpful and allow you to focus on how to apply IA3. If you’re new, we recommend taking a look at the [Translation](https://huggingface.co/docs/transformers/tasks/translation) and [Summarization](https://huggingface.co/docs/transformers/tasks/summarization) guides first from the Transformers documentation. When you’re ready, come back and see how easy it is to drop PEFT in to your training!

</Tip>

## Dataset

You'll use the sentences_allagree subset of the [financial_phrasebank](https://huggingface.co/datasets/financial_phrasebank) dataset. This subset contains financial news with 100% annotator agreement on the sentiment label. Take a look at the [dataset viewer](https://huggingface.co/datasets/financial_phrasebank/viewer/sentences_allagree) for a better idea of the data and sentences you'll be working with.

Load the dataset with the [`~datasets.load_dataset`] function. This subset of the dataset only contains a train split, so use the [`~datasets.train_test_split`] function to create a train and validation split. Create a new `text_label` column so it is easier to understand what the `label` values `0`, `1`, and `2` mean.

```py
from datasets import load_dataset

ds = load_dataset("financial_phrasebank", "sentences_allagree")
ds = ds["train"].train_test_split(test_size=0.1)
ds["validation"] = ds["test"]
del ds["test"]

classes = ds["train"].features["label"].names
ds = ds.map(
    lambda x: {"text_label": [classes[label] for label in x["label"]]},
    batched=True,
    num_proc=1,
)

ds["train"][0]
{'sentence': 'It will be operated by Nokia , and supported by its Nokia NetAct network and service management system .


key: _node_content
string_value:
{"id_": "692a5108-4ae5-486b-a70b-acdcdfff7295", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/ia3", "title": "IA3", "tokens": 2278, "retrieve_doc": true, "source": "peft"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a6b0a0ad-22e1-4a79-b967-4815b61f9236", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/ia3", "title": "IA3", "tokens": 2278, "retrieve_doc": true, "source": "peft"}, "hash": "c1393fd481102f1fef41ff3d4b2f88fbdf1a56a876eaf96f6918e3274864df49", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0a603b7b-2be4-4a4b-991e-9c729bcce42a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/ia3", "title": "IA3", "tokens": 2278, "retrieve_doc": true, "source": "peft"}, "hash": "fd14fc3a1fd3f0c2415c34d0a591c06d8721aa694212f61b461246e21dd0de05", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "aba78db8-f2ec-4f84-bea2-8d04c1d1286f", "node_type": "1", "metadata": {}, "hash": "17d5a057dc0ffa8d111096c7dc30b6c5d5f2334282e86560c2748bd339b71b2c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 853, "end_char_idx": 3982, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
If you’re new, we recommend taking a look at the [Translation](https://huggingface.co/docs/transformers/tasks/translation) and [Summarization](https://huggingface.co/docs/transformers/tasks/summarization) guides first from the Transformers documentation. When you’re ready, come back and see how easy it is to drop PEFT in to your training!

</Tip>

## Dataset

You'll use the sentences_allagree subset of the [financial_phrasebank](https://huggingface.co/datasets/financial_phrasebank) dataset. This subset contains financial news with 100% annotator agreement on the sentiment label. Take a look at the [dataset viewer](https://huggingface.co/datasets/financial_phrasebank/viewer/sentences_allagree) for a better idea of the data and sentences you'll be working with.

Load the dataset with the [`~datasets.load_dataset`] function. This subset of the dataset only contains a train split, so use the [`~datasets.train_test_split`] function to create a train and validation split. Create a new `text_label` column so it is easier to understand what the `label` values `0`, `1`, and `2` mean.

```py
from datasets import load_dataset

ds = load_dataset("financial_phrasebank", "sentences_allagree")
ds = ds["train"].train_test_split(test_size=0.1)
ds["validation"] = ds["test"]
del ds["test"]

classes = ds["train"].features["label"].names
ds = ds.map(
    lambda x: {"text_label": [classes[label] for label in x["label"]]},
    batched=True,
    num_proc=1,
)

ds["train"][0]
{'sentence': 'It will be operated by Nokia , and supported by its Nokia NetAct network and service management system .',
 'label': 1,
 'text_label': 'neutral'}
```

Load a tokenizer and create a preprocessing function that:

1. tokenizes the inputs, pads and truncates the sequence to the `max_length`
2. apply the same tokenizer to the labels but with a shorter `max_length` that corresponds to the label
3. mask the padding tokens

```py
from transformers import AutoTokenizer

text_column = "sentence"
label_column = "text_label"
max_length = 128

tokenizer = AutoTokenizer.from_pretrained("bigscience/mt0-large")

def preprocess_function(examples):
    inputs = examples[text_column]
    targets = examples[label_column]
    model_inputs = tokenizer(inputs, max_length=max_length, padding="max_length", truncation=True, return_tensors="pt")
    labels = tokenizer(targets, max_length=3, padding="max_length", truncation=True, return_tensors="pt")
    labels = labels["input_ids"]
    labels[labels == tokenizer.pad_token_id] = -100
    model_inputs["labels"] = labels
    return model_inputs
```

Use the [`~datasets.Dataset.map`] function to apply the preprocessing function to the entire dataset.

```py
processed_ds = ds.map(
    preprocess_function,
    batched=True,
    num_proc=1,
    remove_columns=ds["train"].column_names,
    load_from_cache_file=False,
    desc="Running tokenizer on dataset",
)
```

Create a training and evaluation [`DataLoader`](https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader), and set `pin_memory=True` to speed up data transfer to the GPU during training if your dataset samples are on a CPU.


key: _node_content
string_value:
{"id_": "aba78db8-f2ec-4f84-bea2-8d04c1d1286f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/ia3", "title": "IA3", "tokens": 2278, "retrieve_doc": true, "source": "peft"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a6b0a0ad-22e1-4a79-b967-4815b61f9236", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/ia3", "title": "IA3", "tokens": 2278, "retrieve_doc": true, "source": "peft"}, "hash": "c1393fd481102f1fef41ff3d4b2f88fbdf1a56a876eaf96f6918e3274864df49", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "692a5108-4ae5-486b-a70b-acdcdfff7295", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/ia3", "title": "IA3", "tokens": 2278, "retrieve_doc": true, "source": "peft"}, "hash": "f4c4202dd0b073daeb50d424f39f9a375fc43e2f9ef36b9acc8a17aa9c6cf297", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "33361189-5142-4d6a-a57d-b900392e3d0c", "node_type": "1", "metadata": {}, "hash": "ccdc10876bddff76b0e2290efcbfa77ccf8e7fcfb750a1f96ab1839f7e18f72b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2447, "end_char_idx": 5467, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
',
 'label': 1,
 'text_label': 'neutral'}
```

Load a tokenizer and create a preprocessing function that:

1. tokenizes the inputs, pads and truncates the sequence to the `max_length`
2. apply the same tokenizer to the labels but with a shorter `max_length` that corresponds to the label
3. mask the padding tokens

```py
from transformers import AutoTokenizer

text_column = "sentence"
label_column = "text_label"
max_length = 128

tokenizer = AutoTokenizer.from_pretrained("bigscience/mt0-large")

def preprocess_function(examples):
    inputs = examples[text_column]
    targets = examples[label_column]
    model_inputs = tokenizer(inputs, max_length=max_length, padding="max_length", truncation=True, return_tensors="pt")
    labels = tokenizer(targets, max_length=3, padding="max_length", truncation=True, return_tensors="pt")
    labels = labels["input_ids"]
    labels[labels == tokenizer.pad_token_id] = -100
    model_inputs["labels"] = labels
    return model_inputs
```

Use the [`~datasets.Dataset.map`] function to apply the preprocessing function to the entire dataset.

```py
processed_ds = ds.map(
    preprocess_function,
    batched=True,
    num_proc=1,
    remove_columns=ds["train"].column_names,
    load_from_cache_file=False,
    desc="Running tokenizer on dataset",
)
```

Create a training and evaluation [`DataLoader`](https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader), and set `pin_memory=True` to speed up data transfer to the GPU during training if your dataset samples are on a CPU.

```py
from torch.utils.data import DataLoader
from transformers import default_data_collator

train_ds = processed_ds["train"]
eval_ds = processed_ds["validation"]

batch_size = 8

train_dataloader = DataLoader(
    train_ds, shuffle=True, collate_fn=default_data_collator, batch_size=batch_size, pin_memory=True
)
eval_dataloader = DataLoader(eval_ds, collate_fn=default_data_collator, batch_size=batch_size, pin_memory=True)
```

## Model

Now you can load a pretrained model to use as the base model for IA3. This guide uses the [bigscience/mt0-large](https://huggingface.co/bigscience/mt0-large) model, but you can use any sequence-to-sequence model you like.

```py
from transformers import AutoModelForSeq2SeqLM

model = AutoModelForSeq2SeqLM.from_pretrained("bigscience/mt0-large")
```

### PEFT configuration and model

All PEFT methods need a configuration that contains and specifies all the parameters for how the PEFT method should be applied. Create an [`IA3Config`] with the task type and set the inference mode to `False`. You can find additional parameters for this configuration in the [API reference](../package_reference/ia3#ia3config).

<Tip>

Call the [`~PeftModel.print_trainable_parameters`] method to compare the number of trainable parameters of [`PeftModel`] versus the number of parameters in the base model!

</Tip>

Once the configuration is setup, pass it to the [`get_peft_model`] function along with the base model to create a trainable [`PeftModel`].


key: _node_content
string_value:
{"id_": "33361189-5142-4d6a-a57d-b900392e3d0c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/ia3", "title": "IA3", "tokens": 2278, "retrieve_doc": true, "source": "peft"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a6b0a0ad-22e1-4a79-b967-4815b61f9236", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/ia3", "title": "IA3", "tokens": 2278, "retrieve_doc": true, "source": "peft"}, "hash": "c1393fd481102f1fef41ff3d4b2f88fbdf1a56a876eaf96f6918e3274864df49", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "aba78db8-f2ec-4f84-bea2-8d04c1d1286f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/ia3", "title": "IA3", "tokens": 2278, "retrieve_doc": true, "source": "peft"}, "hash": "f34c366452687774aeb6506db7f9c5c272541ba1c2c43b8ba0c3b5f0aab9f0ac", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e913d71b-9ba1-411b-8e77-da6ac2369548", "node_type": "1", "metadata": {}, "hash": "684f3721f5f000c72e93abdb2c64e45c0e61c5bdd37bddd0ca0cced01968e6d4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3984, "end_char_idx": 6239, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
from torch.utils.data import DataLoader
from transformers import default_data_collator

train_ds = processed_ds["train"]
eval_ds = processed_ds["validation"]

batch_size = 8

train_dataloader = DataLoader(
    train_ds, shuffle=True, collate_fn=default_data_collator, batch_size=batch_size, pin_memory=True
)
eval_dataloader = DataLoader(eval_ds, collate_fn=default_data_collator, batch_size=batch_size, pin_memory=True)
```

## Model

Now you can load a pretrained model to use as the base model for IA3. This guide uses the [bigscience/mt0-large](https://huggingface.co/bigscience/mt0-large) model, but you can use any sequence-to-sequence model you like.

```py
from transformers import AutoModelForSeq2SeqLM

model = AutoModelForSeq2SeqLM.from_pretrained("bigscience/mt0-large")
```

### PEFT configuration and model

All PEFT methods need a configuration that contains and specifies all the parameters for how the PEFT method should be applied. Create an [`IA3Config`] with the task type and set the inference mode to `False`. You can find additional parameters for this configuration in the [API reference](../package_reference/ia3#ia3config).

<Tip>

Call the [`~PeftModel.print_trainable_parameters`] method to compare the number of trainable parameters of [`PeftModel`] versus the number of parameters in the base model!

</Tip>

Once the configuration is setup, pass it to the [`get_peft_model`] function along with the base model to create a trainable [`PeftModel`].

```py
from peft import IA3Config, get_peft_model

peft_config = IA3Config(task_type="SEQ_2_SEQ_LM")
model = get_peft_model(model, peft_config)
model.print_trainable_parameters()
"trainable params: 282,624 || all params: 1,229,863,936 || trainable%: 0.022980103060766553"
```

### Training

Set up an optimizer and learning rate scheduler.

```py
import torch
from transformers import get_linear_schedule_with_warmup

lr = 8e-3
num_epochs = 3

optimizer = torch.optim.AdamW(model.parameters(), lr=lr)
lr_scheduler = get_linear_schedule_with_warmup(
    optimizer=optimizer,
    num_warmup_steps=0,
    num_training_steps=(len(train_dataloader) * num_epochs),
)
```

Move the model to the GPU and create a training loop that reports the loss and perplexity for each epoch.


key: _node_content
string_value:
{"id_": "e913d71b-9ba1-411b-8e77-da6ac2369548", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/ia3", "title": "IA3", "tokens": 2278, "retrieve_doc": true, "source": "peft"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a6b0a0ad-22e1-4a79-b967-4815b61f9236", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/ia3", "title": "IA3", "tokens": 2278, "retrieve_doc": true, "source": "peft"}, "hash": "c1393fd481102f1fef41ff3d4b2f88fbdf1a56a876eaf96f6918e3274864df49", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "33361189-5142-4d6a-a57d-b900392e3d0c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/ia3", "title": "IA3", "tokens": 2278, "retrieve_doc": true, "source": "peft"}, "hash": "82338673aea5b0201b7657143ac9a94d4fb6193bfdd6661ef198a907d703a479", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1119e336-ce77-4355-b9a3-766238582f28", "node_type": "1", "metadata": {}, "hash": "09fffd659c933e540d8afdcb8f7b7cbb01455de1887e42818ce8864db1a3e12c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4649, "end_char_idx": 7710, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
from transformers import AutoModelForSeq2SeqLM

model = AutoModelForSeq2SeqLM.from_pretrained("bigscience/mt0-large")
```

### PEFT configuration and model

All PEFT methods need a configuration that contains and specifies all the parameters for how the PEFT method should be applied. Create an [`IA3Config`] with the task type and set the inference mode to `False`. You can find additional parameters for this configuration in the [API reference](../package_reference/ia3#ia3config).

<Tip>

Call the [`~PeftModel.print_trainable_parameters`] method to compare the number of trainable parameters of [`PeftModel`] versus the number of parameters in the base model!

</Tip>

Once the configuration is setup, pass it to the [`get_peft_model`] function along with the base model to create a trainable [`PeftModel`].

```py
from peft import IA3Config, get_peft_model

peft_config = IA3Config(task_type="SEQ_2_SEQ_LM")
model = get_peft_model(model, peft_config)
model.print_trainable_parameters()
"trainable params: 282,624 || all params: 1,229,863,936 || trainable%: 0.022980103060766553"
```

### Training

Set up an optimizer and learning rate scheduler.

```py
import torch
from transformers import get_linear_schedule_with_warmup

lr = 8e-3
num_epochs = 3

optimizer = torch.optim.AdamW(model.parameters(), lr=lr)
lr_scheduler = get_linear_schedule_with_warmup(
    optimizer=optimizer,
    num_warmup_steps=0,
    num_training_steps=(len(train_dataloader) * num_epochs),
)
```

Move the model to the GPU and create a training loop that reports the loss and perplexity for each epoch.

```py
from tqdm import tqdm

device = "cuda"
model = model.to(device)

for epoch in range(num_epochs):
    model.train()
    total_loss = 0
    for step, batch in enumerate(tqdm(train_dataloader)):
        batch = {k: v.to(device) for k, v in batch.items()}
        outputs = model(**batch)
        loss = outputs.loss
        total_loss += loss.detach().float()
        loss.backward()
        optimizer.step()
        lr_scheduler.step()
        optimizer.zero_grad()

    model.eval()
    eval_loss = 0
    eval_preds = []
    for step, batch in enumerate(tqdm(eval_dataloader)):
        batch = {k: v.to(device) for k, v in batch.items()}
        with torch.no_grad():
            outputs = model(**batch)
        loss = outputs.loss
        eval_loss += loss.detach().float()
        eval_preds.extend(
            tokenizer.batch_decode(torch.argmax(outputs.logits, -1).detach().cpu().numpy(), skip_special_tokens=True)
        )

    eval_epoch_loss = eval_loss / len(eval_dataloader)
    eval_ppl = torch.exp(eval_epoch_loss)
    train_epoch_loss = total_loss / len(train_dataloader)
    train_ppl = torch.exp(train_epoch_loss)
    print(f"{epoch=}: {train_ppl=} {train_epoch_loss=} {eval_ppl=} {eval_epoch_loss=}")
```

## Share your model

After training is complete, you can upload your model to the Hub with the [`~transformers.PreTrainedModel.push_to_hub`] method. You'll need to login to your Hugging Face account first and enter your token when prompted.


key: _node_content
string_value:
{"id_": "1119e336-ce77-4355-b9a3-766238582f28", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/ia3", "title": "IA3", "tokens": 2278, "retrieve_doc": true, "source": "peft"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a6b0a0ad-22e1-4a79-b967-4815b61f9236", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/ia3", "title": "IA3", "tokens": 2278, "retrieve_doc": true, "source": "peft"}, "hash": "c1393fd481102f1fef41ff3d4b2f88fbdf1a56a876eaf96f6918e3274864df49", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e913d71b-9ba1-411b-8e77-da6ac2369548", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/ia3", "title": "IA3", "tokens": 2278, "retrieve_doc": true, "source": "peft"}, "hash": "67f4dbd4db014cc3d97d25e20e220f74d52b2a809e9c5694499f2bc226293c21", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6241, "end_char_idx": 8907, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
from tqdm import tqdm

device = "cuda"
model = model.to(device)

for epoch in range(num_epochs):
    model.train()
    total_loss = 0
    for step, batch in enumerate(tqdm(train_dataloader)):
        batch = {k: v.to(device) for k, v in batch.items()}
        outputs = model(**batch)
        loss = outputs.loss
        total_loss += loss.detach().float()
        loss.backward()
        optimizer.step()
        lr_scheduler.step()
        optimizer.zero_grad()

    model.eval()
    eval_loss = 0
    eval_preds = []
    for step, batch in enumerate(tqdm(eval_dataloader)):
        batch = {k: v.to(device) for k, v in batch.items()}
        with torch.no_grad():
            outputs = model(**batch)
        loss = outputs.loss
        eval_loss += loss.detach().float()
        eval_preds.extend(
            tokenizer.batch_decode(torch.argmax(outputs.logits, -1).detach().cpu().numpy(), skip_special_tokens=True)
        )

    eval_epoch_loss = eval_loss / len(eval_dataloader)
    eval_ppl = torch.exp(eval_epoch_loss)
    train_epoch_loss = total_loss / len(train_dataloader)
    train_ppl = torch.exp(train_epoch_loss)
    print(f"{epoch=}: {train_ppl=} {train_epoch_loss=} {eval_ppl=} {eval_epoch_loss=}")
```

## Share your model

After training is complete, you can upload your model to the Hub with the [`~transformers.PreTrainedModel.push_to_hub`] method. You'll need to login to your Hugging Face account first and enter your token when prompted.

```py
from huggingface_hub import notebook_login

account = <your-hf-account-name>
peft_model_id = f"{account}/mt0-large-ia3"
model.push_to_hub(peft_model_id)
```

## Inference

To load the model for inference, use the [`~AutoPeftModelForSeq2SeqLM.from_pretrained`] method. Let's also load a sentence of financial news from the dataset to generate a sentiment for.

```py
from peft import AutoPeftModelForSeq2SeqLM

model = AutoPeftModelForSeq2SeqLM.from_pretrained("<your-hf-account-name>/mt0-large-ia3").to("cuda")
tokenizer = AutoTokenizer.from_pretrained("bigscience/mt0-large")

i = 15
inputs = tokenizer(ds["validation"][text_column][i], return_tensors="pt")
print(ds["validation"][text_column][i])
"The robust growth was the result of the inclusion of clothing chain Lindex in the Group in December 2007 ."
```

Call the [`~transformers.GenerationMixin.generate`] method to generate the predicted sentiment label.

```py
with torch.no_grad():
    inputs = {k: v.to(device) for k, v in inputs.items()}
    outputs = model.generate(input_ids=inputs["input_ids"], max_new_tokens=10)
    print(tokenizer.batch_decode(outputs.detach().cpu().numpy(), skip_special_tokens=True))
['positive']
```


key: _node_content
string_value:
{"id_": "16426e66-4aff-4eb7-b8ff-d825f512e333", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/lora_based_methods", "title": "LoRA methods", "tokens": 3714, "retrieve_doc": true, "source": "peft"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8fa7e09a-0308-436d-aadd-1a1a42e7af91", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/lora_based_methods", "title": "LoRA methods", "tokens": 3714, "retrieve_doc": true, "source": "peft"}, "hash": "fdfa2ef49870e6c8a07145f9c11c061ca3407c026afab75000f639c06ba22492", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "45f1b693-63c1-4a22-bcaf-d65b5f41aacf", "node_type": "1", "metadata": {}, "hash": "ee9a75e0d47f903f3b306bd38edf76828f7f9abb73c25179587e27fc7f295773", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3134, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# LoRA methods

A popular way to efficiently train large models is to insert (typically in the attention blocks) smaller trainable matrices that are a low-rank decomposition of the delta weight matrix to be learnt during finetuning. The pretrained model's original weight matrix is frozen and only the smaller matrices are updated during training. This reduces the number of trainable parameters, reducing memory usage and training time which can be very expensive for large models.

There are several different ways to express the weight matrix as a low-rank decomposition, but [Low-Rank Adaptation (LoRA)](../conceptual_guides/adapter#low-rank-adaptation-lora) is the most common method. The PEFT library supports several other LoRA variants, such as [Low-Rank Hadamard Product (LoHa)](../conceptual_guides/adapter#low-rank-hadamard-product-loha), [Low-Rank Kronecker Product (LoKr)](../conceptual_guides/adapter#low-rank-kronecker-product-lokr), and [Adaptive Low-Rank Adaptation (AdaLoRA)](../conceptual_guides/adapter#adaptive-low-rank-adaptation-adalora). You can learn more about how these methods work conceptually in the [Adapters](../conceptual_guides/adapter) guide. If you're interested in applying these methods to other tasks and use cases like semantic segmentation, token classification, take a look at our [notebook collection](https://huggingface.co/collections/PEFT/notebooks-6573b28b33e5a4bf5b157fc1)!

This guide will show you how to quickly train an image classification model - with a low-rank decomposition method - to identify the class of food shown in an image.

<Tip>

Some familiarity with the general process of training an image classification model would be really helpful and allow you to focus on the low-rank decomposition methods. If you're new, we recommend taking a look at the [Image classification](https://huggingface.co/docs/transformers/tasks/image_classification) guide first from the Transformers documentation. When you're ready, come back and see how easy it is to drop PEFT in to your training!

</Tip>

Before you begin, make sure you have all the necessary libraries installed.

```bash
pip install -q peft transformers datasets
```

## Dataset

In this guide, you'll use the [Food-101](https://huggingface.co/datasets/food101) dataset which contains images of 101 food classes (take a look at the [dataset viewer](https://huggingface.co/datasets/food101/viewer/default/train) to get a better idea of what the dataset looks like).

Load the dataset with the [`~datasets.load_dataset`] function.

```py
from datasets import load_dataset

ds = load_dataset("food101")
```

Each food class is labeled with an integer, so to make it easier to understand what these integers represent, you'll create a `label2id` and `id2label` dictionary to map the integer to its class label.

```py
labels = ds["train"].features["label"].names
label2id, id2label = dict(), dict()
for i, label in enumerate(labels):
    label2id[label] = i
    id2label[i] = label

id2label[2]
"baklava"
```

Load an image processor to properly resize and normalize the pixel values of the training and evaluation images.


key: _node_content
string_value:
{"id_": "45f1b693-63c1-4a22-bcaf-d65b5f41aacf", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/lora_based_methods", "title": "LoRA methods", "tokens": 3714, "retrieve_doc": true, "source": "peft"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8fa7e09a-0308-436d-aadd-1a1a42e7af91", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/lora_based_methods", "title": "LoRA methods", "tokens": 3714, "retrieve_doc": true, "source": "peft"}, "hash": "fdfa2ef49870e6c8a07145f9c11c061ca3407c026afab75000f639c06ba22492", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "16426e66-4aff-4eb7-b8ff-d825f512e333", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/lora_based_methods", "title": "LoRA methods", "tokens": 3714, "retrieve_doc": true, "source": "peft"}, "hash": "6ab74ff5aadd0061b8eb485896d56ad2af959b4a8248c414ea8a21f164f3bad3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5a709dd2-b73a-4a42-8e32-1b5069b510e0", "node_type": "1", "metadata": {}, "hash": "6f9b86f39cf38b8a18144ce8f8b4db211e34ff476de99b4c8248ef71ba1faf79", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1423, "end_char_idx": 4777, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This guide will show you how to quickly train an image classification model - with a low-rank decomposition method - to identify the class of food shown in an image.

<Tip>

Some familiarity with the general process of training an image classification model would be really helpful and allow you to focus on the low-rank decomposition methods. If you're new, we recommend taking a look at the [Image classification](https://huggingface.co/docs/transformers/tasks/image_classification) guide first from the Transformers documentation. When you're ready, come back and see how easy it is to drop PEFT in to your training!

</Tip>

Before you begin, make sure you have all the necessary libraries installed.

```bash
pip install -q peft transformers datasets
```

## Dataset

In this guide, you'll use the [Food-101](https://huggingface.co/datasets/food101) dataset which contains images of 101 food classes (take a look at the [dataset viewer](https://huggingface.co/datasets/food101/viewer/default/train) to get a better idea of what the dataset looks like).

Load the dataset with the [`~datasets.load_dataset`] function.

```py
from datasets import load_dataset

ds = load_dataset("food101")
```

Each food class is labeled with an integer, so to make it easier to understand what these integers represent, you'll create a `label2id` and `id2label` dictionary to map the integer to its class label.

```py
labels = ds["train"].features["label"].names
label2id, id2label = dict(), dict()
for i, label in enumerate(labels):
    label2id[label] = i
    id2label[i] = label

id2label[2]
"baklava"
```

Load an image processor to properly resize and normalize the pixel values of the training and evaluation images.

```py
from transformers import AutoImageProcessor

image_processor = AutoImageProcessor.from_pretrained("google/vit-base-patch16-224-in21k")
```

You can also use the image processor to prepare some transformation functions for data augmentation and pixel scaling.

```py
from torchvision.transforms import (
    CenterCrop,
    Compose,
    Normalize,
    RandomHorizontalFlip,
    RandomResizedCrop,
    Resize,
    ToTensor,
)

normalize = Normalize(mean=image_processor.image_mean, std=image_processor.image_std)
train_transforms = Compose(
    [
        RandomResizedCrop(image_processor.size["height"]),
        RandomHorizontalFlip(),
        ToTensor(),
        normalize,
    ]
)

val_transforms = Compose(
    [
        Resize(image_processor.size["height"]),
        CenterCrop(image_processor.size["height"]),
        ToTensor(),
        normalize,
    ]
)

def preprocess_train(example_batch):
    example_batch["pixel_values"] = [train_transforms(image.convert("RGB")) for image in example_batch["image"]]
    return example_batch

def preprocess_val(example_batch):
    example_batch["pixel_values"] = [val_transforms(image.convert("RGB")) for image in example_batch["image"]]
    return example_batch
```

Define the training and validation datasets, and use the [`~datasets.Dataset.set_transform`] function to apply the transformations on-the-fly.

```py
train_ds = ds["train"]
val_ds = ds["validation"]

train_ds.set_transform(preprocess_train)
val_ds.set_transform(preprocess_val)
```

Finally, you'll need a data collator to create a batch of training and evaluation data and convert the labels to `torch.tensor` objects.


key: _node_content
string_value:
{"id_": "5a709dd2-b73a-4a42-8e32-1b5069b510e0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/lora_based_methods", "title": "LoRA methods", "tokens": 3714, "retrieve_doc": true, "source": "peft"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8fa7e09a-0308-436d-aadd-1a1a42e7af91", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/lora_based_methods", "title": "LoRA methods", "tokens": 3714, "retrieve_doc": true, "source": "peft"}, "hash": "fdfa2ef49870e6c8a07145f9c11c061ca3407c026afab75000f639c06ba22492", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "45f1b693-63c1-4a22-bcaf-d65b5f41aacf", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/lora_based_methods", "title": "LoRA methods", "tokens": 3714, "retrieve_doc": true, "source": "peft"}, "hash": "9ec0d63b3a43f40391bdfa7540cd6ff73e255a0eeb1aada088860a6c6fea9cc4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "93791435-be69-4191-9a69-5bfb1054b1fc", "node_type": "1", "metadata": {}, "hash": "5234f433ba166ba12c7d90e5d524f2d18d337218391f23c6552e8f33b14e4181", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3136, "end_char_idx": 6558, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
from transformers import AutoImageProcessor

image_processor = AutoImageProcessor.from_pretrained("google/vit-base-patch16-224-in21k")
```

You can also use the image processor to prepare some transformation functions for data augmentation and pixel scaling.

```py
from torchvision.transforms import (
    CenterCrop,
    Compose,
    Normalize,
    RandomHorizontalFlip,
    RandomResizedCrop,
    Resize,
    ToTensor,
)

normalize = Normalize(mean=image_processor.image_mean, std=image_processor.image_std)
train_transforms = Compose(
    [
        RandomResizedCrop(image_processor.size["height"]),
        RandomHorizontalFlip(),
        ToTensor(),
        normalize,
    ]
)

val_transforms = Compose(
    [
        Resize(image_processor.size["height"]),
        CenterCrop(image_processor.size["height"]),
        ToTensor(),
        normalize,
    ]
)

def preprocess_train(example_batch):
    example_batch["pixel_values"] = [train_transforms(image.convert("RGB")) for image in example_batch["image"]]
    return example_batch

def preprocess_val(example_batch):
    example_batch["pixel_values"] = [val_transforms(image.convert("RGB")) for image in example_batch["image"]]
    return example_batch
```

Define the training and validation datasets, and use the [`~datasets.Dataset.set_transform`] function to apply the transformations on-the-fly.

```py
train_ds = ds["train"]
val_ds = ds["validation"]

train_ds.set_transform(preprocess_train)
val_ds.set_transform(preprocess_val)
```

Finally, you'll need a data collator to create a batch of training and evaluation data and convert the labels to `torch.tensor` objects.

```py
import torch

def collate_fn(examples):
    pixel_values = torch.stack([example["pixel_values"] for example in examples])
    labels = torch.tensor([example["label"] for example in examples])
    return {"pixel_values": pixel_values, "labels": labels}
```

## Model

Now let's load a pretrained model to use as the base model. This guide uses the [google/vit-base-patch16-224-in21k](https://huggingface.co/google/vit-base-patch16-224-in21k) model, but you can use any image classification model you want. Pass the `label2id` and `id2label` dictionaries to the model so it knows how to map the integer labels to their class labels, and you can optionally pass the `ignore_mismatched_sizes=True` parameter if you're finetuning a checkpoint that has already been finetuned.

```py
from transformers import AutoModelForImageClassification, TrainingArguments, Trainer

model = AutoModelForImageClassification.from_pretrained(
    "google/vit-base-patch16-224-in21k",
    label2id=label2id,
    id2label=id2label,
    ignore_mismatched_sizes=True,
)
```

### PEFT configuration and model

Every PEFT method requires a configuration that holds all the parameters specifying how the PEFT method should be applied. Once the configuration is setup, pass it to the [`~peft.get_peft_model`] function along with the base model to create a trainable [`PeftModel`].

<Tip>

Call the [`~PeftModel.print_trainable_parameters`] method to compare the number of parameters of [`PeftModel`] versus the number of parameters in the base model!

</Tip>

<hfoptions id="loras">
<hfoption id="LoRA">

[LoRA](../conceptual_guides/adapter#low-rank-adaptation-lora) decomposes the weight update matrix into *two* smaller matrices. The size of these low-rank matrices is determined by its *rank* or `r`.


key: _node_content
string_value:
{"id_": "93791435-be69-4191-9a69-5bfb1054b1fc", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/lora_based_methods", "title": "LoRA methods", "tokens": 3714, "retrieve_doc": true, "source": "peft"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8fa7e09a-0308-436d-aadd-1a1a42e7af91", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/lora_based_methods", "title": "LoRA methods", "tokens": 3714, "retrieve_doc": true, "source": "peft"}, "hash": "fdfa2ef49870e6c8a07145f9c11c061ca3407c026afab75000f639c06ba22492", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5a709dd2-b73a-4a42-8e32-1b5069b510e0", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/lora_based_methods", "title": "LoRA methods", "tokens": 3714, "retrieve_doc": true, "source": "peft"}, "hash": "277be5c4350f8fea04836a743a0d5ba251d965eca4525b2d7e1b0c1712246e44", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f4739077-7ba0-444b-8a05-d06fa4aaccda", "node_type": "1", "metadata": {}, "hash": "45aaf23d759cb88f411a683547317bb4087060739fd1c7b695f7e89aaa445000", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5112, "end_char_idx": 8159, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This guide uses the [google/vit-base-patch16-224-in21k](https://huggingface.co/google/vit-base-patch16-224-in21k) model, but you can use any image classification model you want. Pass the `label2id` and `id2label` dictionaries to the model so it knows how to map the integer labels to their class labels, and you can optionally pass the `ignore_mismatched_sizes=True` parameter if you're finetuning a checkpoint that has already been finetuned.

```py
from transformers import AutoModelForImageClassification, TrainingArguments, Trainer

model = AutoModelForImageClassification.from_pretrained(
    "google/vit-base-patch16-224-in21k",
    label2id=label2id,
    id2label=id2label,
    ignore_mismatched_sizes=True,
)
```

### PEFT configuration and model

Every PEFT method requires a configuration that holds all the parameters specifying how the PEFT method should be applied. Once the configuration is setup, pass it to the [`~peft.get_peft_model`] function along with the base model to create a trainable [`PeftModel`].

<Tip>

Call the [`~PeftModel.print_trainable_parameters`] method to compare the number of parameters of [`PeftModel`] versus the number of parameters in the base model!

</Tip>

<hfoptions id="loras">
<hfoption id="LoRA">

[LoRA](../conceptual_guides/adapter#low-rank-adaptation-lora) decomposes the weight update matrix into *two* smaller matrices. The size of these low-rank matrices is determined by its *rank* or `r`. A higher rank means the model has more parameters to train, but it also means the model has more learning capacity. You'll also want to specify the `target_modules` which determine where the smaller matrices are inserted. For this guide, you'll target the *query* and *value* matrices of the attention blocks. Other important parameters to set are `lora_alpha` (scaling factor), `bias` (whether `none`, `all` or only the LoRA bias parameters should be trained), and `modules_to_save` (the modules apart from the LoRA layers to be trained and saved). All of these parameters - and more - are found in the [`LoraConfig`].

```py
from peft import LoraConfig, get_peft_model

config = LoraConfig(
    r=16,
    lora_alpha=16,
    target_modules=["query", "value"],
    lora_dropout=0.1,
    bias="none",
    modules_to_save=["classifier"],
)
model = get_peft_model(model, config)
model.print_trainable_parameters()
"trainable params: 667,493 || all params: 86,543,818 || trainable%: 0.7712775047664294"
```

</hfoption>
<hfoption id="LoHa">

[LoHa](../conceptual_guides/adapter#low-rank-hadamard-product-loha) decomposes the weight update matrix into *four* smaller matrices and each pair of smaller matrices is combined with the Hadamard product. This allows the weight update matrix to keep the same number of trainable parameters when compared to LoRA, but with a higher rank (`r^2` for LoHA when compared to `2*r` for LoRA). The size of the smaller matrices is determined by its *rank* or `r`. You'll also want to specify the `target_modules` which determines where the smaller matrices are inserted.


key: _node_content
string_value:
{"id_": "f4739077-7ba0-444b-8a05-d06fa4aaccda", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/lora_based_methods", "title": "LoRA methods", "tokens": 3714, "retrieve_doc": true, "source": "peft"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8fa7e09a-0308-436d-aadd-1a1a42e7af91", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/lora_based_methods", "title": "LoRA methods", "tokens": 3714, "retrieve_doc": true, "source": "peft"}, "hash": "fdfa2ef49870e6c8a07145f9c11c061ca3407c026afab75000f639c06ba22492", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "93791435-be69-4191-9a69-5bfb1054b1fc", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/lora_based_methods", "title": "LoRA methods", "tokens": 3714, "retrieve_doc": true, "source": "peft"}, "hash": "8231ca28863105ef7f47d7a0ae30a6f3f398dbeae433fb53eab09ad2907ae826", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a474d3b9-7671-4d33-a08c-70069fe4e160", "node_type": "1", "metadata": {}, "hash": "dcf4d741c531d0223177fe5e468c8ba699b284d241e578aa7d38203ad84198fa", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6675, "end_char_idx": 9592, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
You'll also want to specify the `target_modules` which determine where the smaller matrices are inserted. For this guide, you'll target the *query* and *value* matrices of the attention blocks. Other important parameters to set are `lora_alpha` (scaling factor), `bias` (whether `none`, `all` or only the LoRA bias parameters should be trained), and `modules_to_save` (the modules apart from the LoRA layers to be trained and saved). All of these parameters - and more - are found in the [`LoraConfig`].

```py
from peft import LoraConfig, get_peft_model

config = LoraConfig(
    r=16,
    lora_alpha=16,
    target_modules=["query", "value"],
    lora_dropout=0.1,
    bias="none",
    modules_to_save=["classifier"],
)
model = get_peft_model(model, config)
model.print_trainable_parameters()
"trainable params: 667,493 || all params: 86,543,818 || trainable%: 0.7712775047664294"
```

</hfoption>
<hfoption id="LoHa">

[LoHa](../conceptual_guides/adapter#low-rank-hadamard-product-loha) decomposes the weight update matrix into *four* smaller matrices and each pair of smaller matrices is combined with the Hadamard product. This allows the weight update matrix to keep the same number of trainable parameters when compared to LoRA, but with a higher rank (`r^2` for LoHA when compared to `2*r` for LoRA). The size of the smaller matrices is determined by its *rank* or `r`. You'll also want to specify the `target_modules` which determines where the smaller matrices are inserted. For this guide, you'll target the *query* and *value* matrices of the attention blocks. Other important parameters to set are `alpha` (scaling factor), and `modules_to_save` (the modules apart from the LoHa layers to be trained and saved). All of these parameters - and more - are found in the [`LoHaConfig`].

```py
from peft import LoHaConfig, get_peft_model

config = LoHaConfig(
    r=16,
    alpha=16,
    target_modules=["query", "value"],
    module_dropout=0.1,
    modules_to_save=["classifier"],
)
model = get_peft_model(model, config)
model.print_trainable_parameters()
"trainable params: 1,257,317 || all params: 87,133,642 || trainable%: 1.4429753779831676"
```

</hfoption>
<hfoption id="LoKr">

[LoKr](../conceptual_guides/adapter#low-rank-kronecker-product-lokr) expresses the weight update matrix as a decomposition of a Kronecker product, creating a block matrix that is able to preserve the rank of the original weight matrix. The size of the smaller matrices are determined by its *rank* or `r`. You'll also want to specify the `target_modules` which determines where the smaller matrices are inserted. For this guide, you'll target the *query* and *value* matrices of the attention blocks. Other important parameters to set are `alpha` (scaling factor), and `modules_to_save` (the modules apart from the LoKr layers to be trained and saved). All of these parameters - and more - are found in the [`LoKrConfig`].


key: _node_content
string_value:
{"id_": "a474d3b9-7671-4d33-a08c-70069fe4e160", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/lora_based_methods", "title": "LoRA methods", "tokens": 3714, "retrieve_doc": true, "source": "peft"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8fa7e09a-0308-436d-aadd-1a1a42e7af91", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/lora_based_methods", "title": "LoRA methods", "tokens": 3714, "retrieve_doc": true, "source": "peft"}, "hash": "fdfa2ef49870e6c8a07145f9c11c061ca3407c026afab75000f639c06ba22492", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f4739077-7ba0-444b-8a05-d06fa4aaccda", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/lora_based_methods", "title": "LoRA methods", "tokens": 3714, "retrieve_doc": true, "source": "peft"}, "hash": "d98939ebb2c92329b94f656203c13c7b0550af98deb546acac08ef65378f140a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a5257baa-bcb0-45ab-8ed8-38964d1babb0", "node_type": "1", "metadata": {}, "hash": "11aa23c961b7f08180d662eafbebecfadb216bb2cd1f5a086e2e5389e97b4319", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8053, "end_char_idx": 10634, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
You'll also want to specify the `target_modules` which determines where the smaller matrices are inserted. For this guide, you'll target the *query* and *value* matrices of the attention blocks. Other important parameters to set are `alpha` (scaling factor), and `modules_to_save` (the modules apart from the LoHa layers to be trained and saved). All of these parameters - and more - are found in the [`LoHaConfig`].

```py
from peft import LoHaConfig, get_peft_model

config = LoHaConfig(
    r=16,
    alpha=16,
    target_modules=["query", "value"],
    module_dropout=0.1,
    modules_to_save=["classifier"],
)
model = get_peft_model(model, config)
model.print_trainable_parameters()
"trainable params: 1,257,317 || all params: 87,133,642 || trainable%: 1.4429753779831676"
```

</hfoption>
<hfoption id="LoKr">

[LoKr](../conceptual_guides/adapter#low-rank-kronecker-product-lokr) expresses the weight update matrix as a decomposition of a Kronecker product, creating a block matrix that is able to preserve the rank of the original weight matrix. The size of the smaller matrices are determined by its *rank* or `r`. You'll also want to specify the `target_modules` which determines where the smaller matrices are inserted. For this guide, you'll target the *query* and *value* matrices of the attention blocks. Other important parameters to set are `alpha` (scaling factor), and `modules_to_save` (the modules apart from the LoKr layers to be trained and saved). All of these parameters - and more - are found in the [`LoKrConfig`].

```py
from peft import LoKrConfig, get_peft_model

config = LoKrConfig(
    r=16,
    alpha=16,
    target_modules=["query", "value"],
    module_dropout=0.1,
    modules_to_save=["classifier"],
)
model = get_peft_model(model, config)
model.print_trainable_parameters()
"trainable params: 116,069 || all params: 87,172,042 || trainable%: 0.13314934162033282"
```

</hfoption>
<hfoption id="AdaLoRA">

[AdaLoRA](../conceptual_guides/adapter#adaptive-low-rank-adaptation-adalora) efficiently manages the LoRA parameter budget by assigning important weight matrices more parameters and pruning less important ones. In contrast, LoRA evenly distributes parameters across all modules. You can control the average desired *rank* or `r` of the matrices, and which modules to apply AdaLoRA to with `target_modules`. Other important parameters to set are `lora_alpha` (scaling factor), and `modules_to_save` (the modules apart from the AdaLoRA layers to be trained and saved). All of these parameters - and more - are found in the [`AdaLoraConfig`].


key: _node_content
string_value:
{"id_": "a5257baa-bcb0-45ab-8ed8-38964d1babb0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/lora_based_methods", "title": "LoRA methods", "tokens": 3714, "retrieve_doc": true, "source": "peft"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8fa7e09a-0308-436d-aadd-1a1a42e7af91", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/lora_based_methods", "title": "LoRA methods", "tokens": 3714, "retrieve_doc": true, "source": "peft"}, "hash": "fdfa2ef49870e6c8a07145f9c11c061ca3407c026afab75000f639c06ba22492", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a474d3b9-7671-4d33-a08c-70069fe4e160", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/lora_based_methods", "title": "LoRA methods", "tokens": 3714, "retrieve_doc": true, "source": "peft"}, "hash": "49fa62d1a42cc38936ae101f9fd6b4ceab27be929a62529080e7518d98c68bd7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8c2152e9-7634-4a5a-9bbd-c4aa664d90c3", "node_type": "1", "metadata": {}, "hash": "5c19c35172336b92d6cd3a0c5d3083e00c246c43d8f4177949b92820a75d2338", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9106, "end_char_idx": 12010, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The size of the smaller matrices are determined by its *rank* or `r`. You'll also want to specify the `target_modules` which determines where the smaller matrices are inserted. For this guide, you'll target the *query* and *value* matrices of the attention blocks. Other important parameters to set are `alpha` (scaling factor), and `modules_to_save` (the modules apart from the LoKr layers to be trained and saved). All of these parameters - and more - are found in the [`LoKrConfig`].

```py
from peft import LoKrConfig, get_peft_model

config = LoKrConfig(
    r=16,
    alpha=16,
    target_modules=["query", "value"],
    module_dropout=0.1,
    modules_to_save=["classifier"],
)
model = get_peft_model(model, config)
model.print_trainable_parameters()
"trainable params: 116,069 || all params: 87,172,042 || trainable%: 0.13314934162033282"
```

</hfoption>
<hfoption id="AdaLoRA">

[AdaLoRA](../conceptual_guides/adapter#adaptive-low-rank-adaptation-adalora) efficiently manages the LoRA parameter budget by assigning important weight matrices more parameters and pruning less important ones. In contrast, LoRA evenly distributes parameters across all modules. You can control the average desired *rank* or `r` of the matrices, and which modules to apply AdaLoRA to with `target_modules`. Other important parameters to set are `lora_alpha` (scaling factor), and `modules_to_save` (the modules apart from the AdaLoRA layers to be trained and saved). All of these parameters - and more - are found in the [`AdaLoraConfig`].

```py
from peft import AdaLoraConfig, get_peft_model

config = AdaLoraConfig(
    r=8,
    init_r=12,
    tinit=200,
    tfinal=1000,
    deltaT=10,
    target_modules=["query", "value"],
    modules_to_save=["classifier"],
)
model = get_peft_model(model, config)
model.print_trainable_parameters()
"trainable params: 520,325 || all params: 87,614,722 || trainable%: 0.5938785036606062"
```

</hfoption>
</hfoptions>

### Training

For training, let's use the [`~transformers.Trainer`] class from Transformers. The [`Trainer`] contains a PyTorch training loop, and when you're ready, call [`~transformers.Trainer.train`] to start training. To customize the training run, configure the training hyperparameters in the [`~transformers.TrainingArguments`] class. With LoRA-like methods, you can afford to use a higher batch size and learning rate.

> [!WARNING]
> AdaLoRA has an [`~AdaLoraModel.update_and_allocate`] method that should be called at each training step to update the parameter budget and mask, otherwise the adaptation step is not performed. This requires writing a custom training loop or subclassing the [`~transformers.Trainer`] to incorporate this method. As an example, take a look at this [custom training loop](https://github.com/huggingface/peft/blob/912ad41e96e03652cabf47522cd876076f7a0c4f/examples/conditional_generation/peft_adalora_seq2seq.py#L120).


key: _node_content
string_value:
{"id_": "8c2152e9-7634-4a5a-9bbd-c4aa664d90c3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/lora_based_methods", "title": "LoRA methods", "tokens": 3714, "retrieve_doc": true, "source": "peft"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8fa7e09a-0308-436d-aadd-1a1a42e7af91", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/lora_based_methods", "title": "LoRA methods", "tokens": 3714, "retrieve_doc": true, "source": "peft"}, "hash": "fdfa2ef49870e6c8a07145f9c11c061ca3407c026afab75000f639c06ba22492", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a5257baa-bcb0-45ab-8ed8-38964d1babb0", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/lora_based_methods", "title": "LoRA methods", "tokens": 3714, "retrieve_doc": true, "source": "peft"}, "hash": "09c032d3688940a19ba084aa34caaa4874bbaccba01834303b096343e17474ac", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4bb718aa-b33f-403f-988c-1f2906b095c1", "node_type": "1", "metadata": {}, "hash": "18d6455c80f8ba3a0c8da875d6e060f26dbbaf8689c1e657aa18120fba066376", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10562, "end_char_idx": 13355, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
All of these parameters - and more - are found in the [`AdaLoraConfig`].

```py
from peft import AdaLoraConfig, get_peft_model

config = AdaLoraConfig(
    r=8,
    init_r=12,
    tinit=200,
    tfinal=1000,
    deltaT=10,
    target_modules=["query", "value"],
    modules_to_save=["classifier"],
)
model = get_peft_model(model, config)
model.print_trainable_parameters()
"trainable params: 520,325 || all params: 87,614,722 || trainable%: 0.5938785036606062"
```

</hfoption>
</hfoptions>

### Training

For training, let's use the [`~transformers.Trainer`] class from Transformers. The [`Trainer`] contains a PyTorch training loop, and when you're ready, call [`~transformers.Trainer.train`] to start training. To customize the training run, configure the training hyperparameters in the [`~transformers.TrainingArguments`] class. With LoRA-like methods, you can afford to use a higher batch size and learning rate.

> [!WARNING]
> AdaLoRA has an [`~AdaLoraModel.update_and_allocate`] method that should be called at each training step to update the parameter budget and mask, otherwise the adaptation step is not performed. This requires writing a custom training loop or subclassing the [`~transformers.Trainer`] to incorporate this method. As an example, take a look at this [custom training loop](https://github.com/huggingface/peft/blob/912ad41e96e03652cabf47522cd876076f7a0c4f/examples/conditional_generation/peft_adalora_seq2seq.py#L120).

```py
from transformers import TrainingArguments, Trainer

account = "stevhliu"
peft_model_id = f"{account}/google/vit-base-patch16-224-in21k-lora"
batch_size = 128

args = TrainingArguments(
    peft_model_id,
    remove_unused_columns=False,
    evaluation_strategy="epoch",
    save_strategy="epoch",
    learning_rate=5e-3,
    per_device_train_batch_size=batch_size,
    gradient_accumulation_steps=4,
    per_device_eval_batch_size=batch_size,
    fp16=True,
    num_train_epochs=5,
    logging_steps=10,
    load_best_model_at_end=True,
    label_names=["labels"],
)
```

Begin training with [`~transformers.Trainer.train`].

```py
trainer = Trainer(
    model,
    args,
    train_dataset=train_ds,
    eval_dataset=val_ds,
    tokenizer=image_processor,
    data_collator=collate_fn,
)
trainer.train()
```

## Share your model

Once training is complete, you can upload your model to the Hub with the [`~transformers.PreTrainedModel.push_to_hub`] method. You’ll need to login to your Hugging Face account first and enter your token when prompted.

```py
from huggingface_hub import notebook_login

notebook_login()
```

Call [`~transformers.PreTrainedModel.push_to_hub`] to save your model to your repositoy.

```py
model.push_to_hub(peft_model_id)
```

## Inference

Let's load the model from the Hub and test it out on a food image.


key: _node_content
string_value:
{"id_": "4bb718aa-b33f-403f-988c-1f2906b095c1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/lora_based_methods", "title": "LoRA methods", "tokens": 3714, "retrieve_doc": true, "source": "peft"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8fa7e09a-0308-436d-aadd-1a1a42e7af91", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/lora_based_methods", "title": "LoRA methods", "tokens": 3714, "retrieve_doc": true, "source": "peft"}, "hash": "fdfa2ef49870e6c8a07145f9c11c061ca3407c026afab75000f639c06ba22492", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8c2152e9-7634-4a5a-9bbd-c4aa664d90c3", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/lora_based_methods", "title": "LoRA methods", "tokens": 3714, "retrieve_doc": true, "source": "peft"}, "hash": "27a8a65fe48e043d2787e16a4aed6e4dc714cdbc95522472b01931c5295f3f9b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12012, "end_char_idx": 14566, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
from transformers import TrainingArguments, Trainer

account = "stevhliu"
peft_model_id = f"{account}/google/vit-base-patch16-224-in21k-lora"
batch_size = 128

args = TrainingArguments(
    peft_model_id,
    remove_unused_columns=False,
    evaluation_strategy="epoch",
    save_strategy="epoch",
    learning_rate=5e-3,
    per_device_train_batch_size=batch_size,
    gradient_accumulation_steps=4,
    per_device_eval_batch_size=batch_size,
    fp16=True,
    num_train_epochs=5,
    logging_steps=10,
    load_best_model_at_end=True,
    label_names=["labels"],
)
```

Begin training with [`~transformers.Trainer.train`].

```py
trainer = Trainer(
    model,
    args,
    train_dataset=train_ds,
    eval_dataset=val_ds,
    tokenizer=image_processor,
    data_collator=collate_fn,
)
trainer.train()
```

## Share your model

Once training is complete, you can upload your model to the Hub with the [`~transformers.PreTrainedModel.push_to_hub`] method. You’ll need to login to your Hugging Face account first and enter your token when prompted.

```py
from huggingface_hub import notebook_login

notebook_login()
```

Call [`~transformers.PreTrainedModel.push_to_hub`] to save your model to your repositoy.

```py
model.push_to_hub(peft_model_id)
```

## Inference

Let's load the model from the Hub and test it out on a food image.

```py
from peft import PeftConfig, PeftModel
from transformers import AutoImageProcessor
from PIL import Image
import requests

config = PeftConfig.from_pretrained("stevhliu/vit-base-patch16-224-in21k-lora")
model = AutoModelForImageClassification.from_pretrained(
    config.base_model_name_or_path,
    label2id=label2id,
    id2label=id2label,
    ignore_mismatched_sizes=True,
)
model = PeftModel.from_pretrained(model, "stevhliu/vit-base-patch16-224-in21k-lora")

url = "https://huggingface.co/datasets/sayakpaul/sample-datasets/resolve/main/beignets.jpeg"
image = Image.open(requests.get(url, stream=True).raw)
image
```

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/sayakpaul/sample-datasets/resolve/main/beignets.jpeg">
</div>

Convert the image to RGB and return the underlying PyTorch tensors.

```py
encoding = image_processor(image.convert("RGB"), return_tensors="pt")
```

Now run the model and return the predicted class!

```py
with torch.no_grad():
    outputs = model(**encoding)
    logits = outputs.logits

predicted_class_idx = logits.argmax(-1).item()
print("Predicted class:", model.config.id2label[predicted_class_idx])
"Predicted class: beignets"
```


key: _node_content
string_value:
{"id_": "bee1d7a0-5399-465a-ae5d-75c4983a0b58", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/prompt_based_methods", "title": "Prompt-based methods", "tokens": 3315, "retrieve_doc": true, "source": "peft"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5abc71a2-6dbd-4e86-8702-af415be1c042", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/prompt_based_methods", "title": "Prompt-based methods", "tokens": 3315, "retrieve_doc": true, "source": "peft"}, "hash": "a4974d540eeb655582d44b88d415425379b0d06f7d5e927999502cad9d07e2c7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "10384565-c689-4241-9d17-b97319126851", "node_type": "1", "metadata": {}, "hash": "4c2cd4a4d76068ebb3dc2055c6672419137f99c5dc1781694f5f14a29091a392", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3269, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Prompt-based methods

A prompt can describe a task or provide an example of a task you want the model to learn. Instead of manually creating these prompts, soft prompting methods add learnable parameters to the input embeddings that can be optimized for a specific task while keeping the pretrained model's parameters frozen. This makes it both faster and easier to finetune large language models (LLMs) for new downstream tasks.

The PEFT library supports several types of prompting methods (p-tuning, prefix tuning, prompt tuning) and you can learn more about how these methods work conceptually in the [Soft prompts](../conceptual_guides/prompting) guide. If you're interested in applying these methods to other tasks and use cases, take a look at our [notebook collection](https://huggingface.co/spaces/PEFT/soft-prompting)!

This guide will show you how to train a causal language model - with a soft prompting method - to *generate a classification* for whether a tweet is a complaint or not.

<Tip>

Some familiarity with the general process of training a causal language model would be really helpful and allow you to focus on the soft prompting methods. If you're new, we recommend taking a look at the [Causal language modeling](https://huggingface.co/docs/transformers/tasks/language_modeling) guide first from the Transformers documentation. When you're ready, come back and see how easy it is to drop PEFT in to your training!

</Tip>

Before you begin, make sure you have all the necessary libraries installed.

```bash
pip install -q peft transformers datasets
```

## Dataset

For this guide, you'll use the `twitter_complaints` subset of the [RAFT](https://huggingface.co/datasets/ought/raft) dataset. The `twitter_complaints` subset contains tweets labeled as `complaint` and `no complaint` and you can check out the [dataset viewer](https://huggingface.co/datasets/ought/raft/viewer/twitter_complaints) for a better idea of what the data looks like.

Use the [`~datasets.load_dataset`] function to load the dataset and create a new `text_label` column so it is easier to understand what the `Label` values, `1` and `2` mean.

```py
from datasets import load_dataset

ds = load_dataset("ought/raft", "twitter_complaints")

classes = [k.replace("_", " ") for k in ds["train"].features["Label"].names]
ds = ds.map(
    lambda x: {"text_label": [classes[label] for label in x["Label"]]},
    batched=True,
    num_proc=1,
)
ds["train"][0]
{"Tweet text": "@HMRCcustomers No this is my first job", "ID": 0, "Label": 2, "text_label": "no complaint"}
```

Load a tokenizer, define the padding token to use, and determine the maximum length of the tokenized label.

```py
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bigscience/bloomz-560m")
if tokenizer.pad_token_id is None:
    tokenizer.pad_token_id = tokenizer.eos_token_id
target_max_length = max([len(tokenizer(class_label)["input_ids"]) for class_label in classes])
print(target_max_length)
```

Create a preprocessing function that tokenizes the tweet text and labels, pad the inputs and labels in each batch, create an attention mask, and truncate sequences to the `max_length`. Then convert the `input_ids`, `attention_mask`, and `labels` to PyTorch tensors.


key: _node_content
string_value:
{"id_": "10384565-c689-4241-9d17-b97319126851", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/prompt_based_methods", "title": "Prompt-based methods", "tokens": 3315, "retrieve_doc": true, "source": "peft"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5abc71a2-6dbd-4e86-8702-af415be1c042", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/prompt_based_methods", "title": "Prompt-based methods", "tokens": 3315, "retrieve_doc": true, "source": "peft"}, "hash": "a4974d540eeb655582d44b88d415425379b0d06f7d5e927999502cad9d07e2c7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bee1d7a0-5399-465a-ae5d-75c4983a0b58", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/prompt_based_methods", "title": "Prompt-based methods", "tokens": 3315, "retrieve_doc": true, "source": "peft"}, "hash": "5bc59bca928d8806ff532178cb654dc177791f1f7b201d826db18194381f8e20", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "cbaa1eed-f645-4a41-9260-23e2491161ae", "node_type": "1", "metadata": {}, "hash": "376692206b04986c6f6243510992e45f6e657a8e896b226dcda1da293398f12f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1721, "end_char_idx": 4775, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The `twitter_complaints` subset contains tweets labeled as `complaint` and `no complaint` and you can check out the [dataset viewer](https://huggingface.co/datasets/ought/raft/viewer/twitter_complaints) for a better idea of what the data looks like.

Use the [`~datasets.load_dataset`] function to load the dataset and create a new `text_label` column so it is easier to understand what the `Label` values, `1` and `2` mean.

```py
from datasets import load_dataset

ds = load_dataset("ought/raft", "twitter_complaints")

classes = [k.replace("_", " ") for k in ds["train"].features["Label"].names]
ds = ds.map(
    lambda x: {"text_label": [classes[label] for label in x["Label"]]},
    batched=True,
    num_proc=1,
)
ds["train"][0]
{"Tweet text": "@HMRCcustomers No this is my first job", "ID": 0, "Label": 2, "text_label": "no complaint"}
```

Load a tokenizer, define the padding token to use, and determine the maximum length of the tokenized label.

```py
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bigscience/bloomz-560m")
if tokenizer.pad_token_id is None:
    tokenizer.pad_token_id = tokenizer.eos_token_id
target_max_length = max([len(tokenizer(class_label)["input_ids"]) for class_label in classes])
print(target_max_length)
```

Create a preprocessing function that tokenizes the tweet text and labels, pad the inputs and labels in each batch, create an attention mask, and truncate sequences to the `max_length`. Then convert the `input_ids`, `attention_mask`, and `labels` to PyTorch tensors.

```py
import torch

max_length = 64

def preprocess_function(examples, text_column="Tweet text", label_column="text_label"):
    batch_size = len(examples[text_column])
    inputs = [f"{text_column} : {x} Label : " for x in examples[text_column]]
    targets = [str(x) for x in examples[label_column]]
    model_inputs = tokenizer(inputs)
    labels = tokenizer(targets)
    classes = [k.replace("_", " ") for k in ds["train"].features["Label"].names]
    for i in range(batch_size):
        sample_input_ids = model_inputs["input_ids"][i]
        label_input_ids = labels["input_ids"][i]
        model_inputs["input_ids"][i] = [tokenizer.pad_token_id] * (
            max_length - len(sample_input_ids)
        ) + sample_input_ids
        model_inputs["attention_mask"][i] = [0] * (max_length - len(sample_input_ids)) + model_inputs[
            "attention_mask"
        ][i]
        labels["input_ids"][i] = [-100] * (max_length - len(label_input_ids)) + label_input_ids
        model_inputs["input_ids"][i] = torch.tensor(model_inputs["input_ids"][i][:max_length])
        model_inputs["attention_mask"][i] = torch.tensor(model_inputs["attention_mask"][i][:max_length])
        labels["input_ids"][i] = torch.tensor(labels["input_ids"][i][:max_length])
    model_inputs["labels"] = labels["input_ids"]
    return model_inputs
```

Apply the preprocessing function to the entire dataset with the [`~datasets.Dataset.map`] function, and remove the unprocessed columns because the model won't need them.


key: _node_content
string_value:
{"id_": "cbaa1eed-f645-4a41-9260-23e2491161ae", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/prompt_based_methods", "title": "Prompt-based methods", "tokens": 3315, "retrieve_doc": true, "source": "peft"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5abc71a2-6dbd-4e86-8702-af415be1c042", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/prompt_based_methods", "title": "Prompt-based methods", "tokens": 3315, "retrieve_doc": true, "source": "peft"}, "hash": "a4974d540eeb655582d44b88d415425379b0d06f7d5e927999502cad9d07e2c7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "10384565-c689-4241-9d17-b97319126851", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/prompt_based_methods", "title": "Prompt-based methods", "tokens": 3315, "retrieve_doc": true, "source": "peft"}, "hash": "4e1b04cab24830e861bfc901605792388eb090043b6d5ead117b24c3c36af659", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "735a6d5b-7f99-4b6c-8d67-5905e61d48a2", "node_type": "1", "metadata": {}, "hash": "a8beb1fae6e98477183ff2817bbdb4efbacc3c28fae9e5bd6c18a4784d856b71", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3189, "end_char_idx": 6374, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Then convert the `input_ids`, `attention_mask`, and `labels` to PyTorch tensors.

```py
import torch

max_length = 64

def preprocess_function(examples, text_column="Tweet text", label_column="text_label"):
    batch_size = len(examples[text_column])
    inputs = [f"{text_column} : {x} Label : " for x in examples[text_column]]
    targets = [str(x) for x in examples[label_column]]
    model_inputs = tokenizer(inputs)
    labels = tokenizer(targets)
    classes = [k.replace("_", " ") for k in ds["train"].features["Label"].names]
    for i in range(batch_size):
        sample_input_ids = model_inputs["input_ids"][i]
        label_input_ids = labels["input_ids"][i]
        model_inputs["input_ids"][i] = [tokenizer.pad_token_id] * (
            max_length - len(sample_input_ids)
        ) + sample_input_ids
        model_inputs["attention_mask"][i] = [0] * (max_length - len(sample_input_ids)) + model_inputs[
            "attention_mask"
        ][i]
        labels["input_ids"][i] = [-100] * (max_length - len(label_input_ids)) + label_input_ids
        model_inputs["input_ids"][i] = torch.tensor(model_inputs["input_ids"][i][:max_length])
        model_inputs["attention_mask"][i] = torch.tensor(model_inputs["attention_mask"][i][:max_length])
        labels["input_ids"][i] = torch.tensor(labels["input_ids"][i][:max_length])
    model_inputs["labels"] = labels["input_ids"]
    return model_inputs
```

Apply the preprocessing function to the entire dataset with the [`~datasets.Dataset.map`] function, and remove the unprocessed columns because the model won't need them.

```py
processed_ds = ds.map(
    preprocess_function,
    batched=True,
    num_proc=1,
    remove_columns=ds["train"].column_names,
    load_from_cache_file=False,
    desc="Running tokenizer on dataset",
)
```

Finally, create a training and evaluation [`DataLoader`](https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader). You can set `pin_memory=True` to speed up the data transfer to the GPU during training if the samples in your dataset are on a CPU.

```py
from torch.utils.data import DataLoader
from transformers import default_data_collator

train_ds = processed_ds["train"]
eval_ds = processed_ds["test"]

batch_size = 16

train_dataloader = DataLoader(train_ds, shuffle=True, collate_fn=default_data_collator, batch_size=batch_size, pin_memory=True)
eval_dataloader = DataLoader(eval_ds, collate_fn=default_data_collator, batch_size=batch_size, pin_memory=True)
```

## Model

Now let's load a pretrained model to use as the base model for the soft prompt method. This guide uses the [bigscience/bloomz-560m](https://huggingface.co/bigscience/bloomz-560m) model, but you can use any causal language model you want.

```py
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("bigscience/bloomz-560m")
```

### PEFT configuration and model

For any PEFT method, you'll need to create a configuration which contains all the parameters that specify how the PEFT method should be applied. Once the configuration is setup, pass it to the [`~peft.get_peft_model`] function along with the base model to create a trainable [`PeftModel`].


key: _node_content
string_value:
{"id_": "735a6d5b-7f99-4b6c-8d67-5905e61d48a2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/prompt_based_methods", "title": "Prompt-based methods", "tokens": 3315, "retrieve_doc": true, "source": "peft"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5abc71a2-6dbd-4e86-8702-af415be1c042", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/prompt_based_methods", "title": "Prompt-based methods", "tokens": 3315, "retrieve_doc": true, "source": "peft"}, "hash": "a4974d540eeb655582d44b88d415425379b0d06f7d5e927999502cad9d07e2c7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "cbaa1eed-f645-4a41-9260-23e2491161ae", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/prompt_based_methods", "title": "Prompt-based methods", "tokens": 3315, "retrieve_doc": true, "source": "peft"}, "hash": "9626c92ef0347e94fd5e84a1ae191c30a696579d34154994314d257e0a331094", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9984d8a2-2398-4f2c-920d-06365ad7fea0", "node_type": "1", "metadata": {}, "hash": "98a8952586c8e21b77efe5436ccb0779dfa9dd309b8d05f1d6240454f9ed85a9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4777, "end_char_idx": 7603, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
processed_ds = ds.map(
    preprocess_function,
    batched=True,
    num_proc=1,
    remove_columns=ds["train"].column_names,
    load_from_cache_file=False,
    desc="Running tokenizer on dataset",
)
```

Finally, create a training and evaluation [`DataLoader`](https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader). You can set `pin_memory=True` to speed up the data transfer to the GPU during training if the samples in your dataset are on a CPU.

```py
from torch.utils.data import DataLoader
from transformers import default_data_collator

train_ds = processed_ds["train"]
eval_ds = processed_ds["test"]

batch_size = 16

train_dataloader = DataLoader(train_ds, shuffle=True, collate_fn=default_data_collator, batch_size=batch_size, pin_memory=True)
eval_dataloader = DataLoader(eval_ds, collate_fn=default_data_collator, batch_size=batch_size, pin_memory=True)
```

## Model

Now let's load a pretrained model to use as the base model for the soft prompt method. This guide uses the [bigscience/bloomz-560m](https://huggingface.co/bigscience/bloomz-560m) model, but you can use any causal language model you want.

```py
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("bigscience/bloomz-560m")
```

### PEFT configuration and model

For any PEFT method, you'll need to create a configuration which contains all the parameters that specify how the PEFT method should be applied. Once the configuration is setup, pass it to the [`~peft.get_peft_model`] function along with the base model to create a trainable [`PeftModel`].

<Tip>

Call the [`~PeftModel.print_trainable_parameters`] method to compare the number of trainable parameters of [`PeftModel`] versus the number of parameters in the base model!

</Tip>

<hfoptions id="configurations">
<hfoption id="p-tuning">

[P-tuning](../conceptual_guides/prompting#p-tuning) adds a trainable embedding tensor where the prompt tokens can be added anywhere in the input sequence. Create a [`PromptEncoderConfig`] with the task type, the number of virtual tokens to add and learn, and the hidden size of the encoder for learning the prompt parameters.

```py
from peft import PromptEncoderConfig, get_peft_model

peft_config = PromptEncoderConfig(task_type="CAUSAL_LM", num_virtual_tokens=20, encoder_hidden_size=128)
model = get_peft_model(model, peft_config)
model.print_trainable_parameters()
"trainable params: 300,288 || all params: 559,514,880 || trainable%: 0.05366935013417338"
```

</hfoption>
<hfoption id="prefix tuning">

[Prefix tuning](../conceptual_guides/prompting#prefix-tuning) adds task-specific parameters in all of the model layers, which are optimized by a separate feed-forward network. Create a [`PrefixTuningConfig`] with the task type and number of virtual tokens to add and learn.


key: _node_content
string_value:
{"id_": "9984d8a2-2398-4f2c-920d-06365ad7fea0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/prompt_based_methods", "title": "Prompt-based methods", "tokens": 3315, "retrieve_doc": true, "source": "peft"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5abc71a2-6dbd-4e86-8702-af415be1c042", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/prompt_based_methods", "title": "Prompt-based methods", "tokens": 3315, "retrieve_doc": true, "source": "peft"}, "hash": "a4974d540eeb655582d44b88d415425379b0d06f7d5e927999502cad9d07e2c7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "735a6d5b-7f99-4b6c-8d67-5905e61d48a2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/prompt_based_methods", "title": "Prompt-based methods", "tokens": 3315, "retrieve_doc": true, "source": "peft"}, "hash": "3bb2f64c1a783dd34304abbd4f94e339924dd3da7c995bcf5099a9067030c71d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d6d09e50-de6e-4e15-bd8f-685c031a3b73", "node_type": "1", "metadata": {}, "hash": "49ac478c1e78e5a26434145b27e88748973b6570563f87873588756f129e80ec", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6230, "end_char_idx": 8683, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Once the configuration is setup, pass it to the [`~peft.get_peft_model`] function along with the base model to create a trainable [`PeftModel`].

<Tip>

Call the [`~PeftModel.print_trainable_parameters`] method to compare the number of trainable parameters of [`PeftModel`] versus the number of parameters in the base model!

</Tip>

<hfoptions id="configurations">
<hfoption id="p-tuning">

[P-tuning](../conceptual_guides/prompting#p-tuning) adds a trainable embedding tensor where the prompt tokens can be added anywhere in the input sequence. Create a [`PromptEncoderConfig`] with the task type, the number of virtual tokens to add and learn, and the hidden size of the encoder for learning the prompt parameters.

```py
from peft import PromptEncoderConfig, get_peft_model

peft_config = PromptEncoderConfig(task_type="CAUSAL_LM", num_virtual_tokens=20, encoder_hidden_size=128)
model = get_peft_model(model, peft_config)
model.print_trainable_parameters()
"trainable params: 300,288 || all params: 559,514,880 || trainable%: 0.05366935013417338"
```

</hfoption>
<hfoption id="prefix tuning">

[Prefix tuning](../conceptual_guides/prompting#prefix-tuning) adds task-specific parameters in all of the model layers, which are optimized by a separate feed-forward network. Create a [`PrefixTuningConfig`] with the task type and number of virtual tokens to add and learn.

```py
from peft import PrefixTuningConfig, get_peft_model

peft_config = PrefixTuningConfig(task_type="CAUSAL_LM", num_virtual_tokens=20)
model = get_peft_model(model, peft_config)
model.print_trainable_parameters()
"trainable params: 983,040 || all params: 560,197,632 || trainable%: 0.1754809274167014"
```

</hfoption>
<hfoption id="prompt tuning">

[Prompt tuning](../conceptual_guides/prompting#prompt-tuning) formulates all tasks as a *generation* task and it adds a task-specific prompt to the input which is updated independently. The `prompt_tuning_init_text` parameter specifies how to finetune the model (in this case, it is classifying whether tweets are complaints or not). For the best results, the `prompt_tuning_init_text` should have the same number of tokens that should be predicted. To do this, you can set `num_virtual_tokens` to the number of tokens of the `prompt_tuning_init_text`.

Create a [`PromptTuningConfig`] with the task type, the initial prompt tuning text to train the model with, the number of virtual tokens to add and learn, and a tokenizer.


key: _node_content
string_value:
{"id_": "d6d09e50-de6e-4e15-bd8f-685c031a3b73", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/prompt_based_methods", "title": "Prompt-based methods", "tokens": 3315, "retrieve_doc": true, "source": "peft"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5abc71a2-6dbd-4e86-8702-af415be1c042", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/prompt_based_methods", "title": "Prompt-based methods", "tokens": 3315, "retrieve_doc": true, "source": "peft"}, "hash": "a4974d540eeb655582d44b88d415425379b0d06f7d5e927999502cad9d07e2c7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9984d8a2-2398-4f2c-920d-06365ad7fea0", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/prompt_based_methods", "title": "Prompt-based methods", "tokens": 3315, "retrieve_doc": true, "source": "peft"}, "hash": "ea38bbcd39c04d2a3b60f3a6914945e159aee7d9ef36c8f95d31bc0bd164a339", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d2405782-9a0b-42bd-9fd1-f10a3518f0cd", "node_type": "1", "metadata": {}, "hash": "ea34d3af10f0bc11557d802eeb7e06d5dc7e11924cd290f2cad236c948dcc913", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7506, "end_char_idx": 9819, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Create a [`PrefixTuningConfig`] with the task type and number of virtual tokens to add and learn.

```py
from peft import PrefixTuningConfig, get_peft_model

peft_config = PrefixTuningConfig(task_type="CAUSAL_LM", num_virtual_tokens=20)
model = get_peft_model(model, peft_config)
model.print_trainable_parameters()
"trainable params: 983,040 || all params: 560,197,632 || trainable%: 0.1754809274167014"
```

</hfoption>
<hfoption id="prompt tuning">

[Prompt tuning](../conceptual_guides/prompting#prompt-tuning) formulates all tasks as a *generation* task and it adds a task-specific prompt to the input which is updated independently. The `prompt_tuning_init_text` parameter specifies how to finetune the model (in this case, it is classifying whether tweets are complaints or not). For the best results, the `prompt_tuning_init_text` should have the same number of tokens that should be predicted. To do this, you can set `num_virtual_tokens` to the number of tokens of the `prompt_tuning_init_text`.

Create a [`PromptTuningConfig`] with the task type, the initial prompt tuning text to train the model with, the number of virtual tokens to add and learn, and a tokenizer.

```py
from peft import PromptTuningConfig, PromptTuningInit, get_peft_model

prompt_tuning_init_text = "Classify if the tweet is a complaint or no complaint.\n"
peft_config = PromptTuningConfig(
    task_type="CAUSAL_LM",
    prompt_tuning_init=PromptTuningInit.TEXT,
    num_virtual_tokens=len(tokenizer(prompt_tuning_init_text)["input_ids"]),
    prompt_tuning_init_text=prompt_tuning_init_text,
    tokenizer_name_or_path="bigscience/bloomz-560m",
)
model = get_peft_model(model, peft_config)
model.print_trainable_parameters()
"trainable params: 8,192 || all params: 559,222,784 || trainable%: 0.0014648902430985358"
```

</hfoption>
</hfoptions>

### Training

Set up an optimizer and learning rate scheduler.

```py
from transformers import get_linear_schedule_with_warmup

lr = 3e-2
num_epochs = 50

optimizer = torch.optim.AdamW(model.parameters(), lr=lr)
lr_scheduler = get_linear_schedule_with_warmup(
    optimizer=optimizer,
    num_warmup_steps=0,
    num_training_steps=(len(train_dataloader) * num_epochs),
)
```

Move the model to the GPU and create a training loop that reports the loss and perplexity for each epoch.


key: _node_content
string_value:
{"id_": "d2405782-9a0b-42bd-9fd1-f10a3518f0cd", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/prompt_based_methods", "title": "Prompt-based methods", "tokens": 3315, "retrieve_doc": true, "source": "peft"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5abc71a2-6dbd-4e86-8702-af415be1c042", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/prompt_based_methods", "title": "Prompt-based methods", "tokens": 3315, "retrieve_doc": true, "source": "peft"}, "hash": "a4974d540eeb655582d44b88d415425379b0d06f7d5e927999502cad9d07e2c7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d6d09e50-de6e-4e15-bd8f-685c031a3b73", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/prompt_based_methods", "title": "Prompt-based methods", "tokens": 3315, "retrieve_doc": true, "source": "peft"}, "hash": "7b5bb5b2b66c31f4fdd5c72f0bef3446c522e47d7301333448ec76635d217bff", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "41e713e2-0188-45b1-a2ec-df4c73d17ccf", "node_type": "1", "metadata": {}, "hash": "4b76fd57498d1d04440f80ba6442bcebe3c0188ba37f2d939ff08dc92be03fa8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8292, "end_char_idx": 11289, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
For the best results, the `prompt_tuning_init_text` should have the same number of tokens that should be predicted. To do this, you can set `num_virtual_tokens` to the number of tokens of the `prompt_tuning_init_text`.

Create a [`PromptTuningConfig`] with the task type, the initial prompt tuning text to train the model with, the number of virtual tokens to add and learn, and a tokenizer.

```py
from peft import PromptTuningConfig, PromptTuningInit, get_peft_model

prompt_tuning_init_text = "Classify if the tweet is a complaint or no complaint.\n"
peft_config = PromptTuningConfig(
    task_type="CAUSAL_LM",
    prompt_tuning_init=PromptTuningInit.TEXT,
    num_virtual_tokens=len(tokenizer(prompt_tuning_init_text)["input_ids"]),
    prompt_tuning_init_text=prompt_tuning_init_text,
    tokenizer_name_or_path="bigscience/bloomz-560m",
)
model = get_peft_model(model, peft_config)
model.print_trainable_parameters()
"trainable params: 8,192 || all params: 559,222,784 || trainable%: 0.0014648902430985358"
```

</hfoption>
</hfoptions>

### Training

Set up an optimizer and learning rate scheduler.

```py
from transformers import get_linear_schedule_with_warmup

lr = 3e-2
num_epochs = 50

optimizer = torch.optim.AdamW(model.parameters(), lr=lr)
lr_scheduler = get_linear_schedule_with_warmup(
    optimizer=optimizer,
    num_warmup_steps=0,
    num_training_steps=(len(train_dataloader) * num_epochs),
)
```

Move the model to the GPU and create a training loop that reports the loss and perplexity for each epoch.

```py
from tqdm import tqdm

device = "cuda"
model = model.to(device)

for epoch in range(num_epochs):
    model.train()
    total_loss = 0
    for step, batch in enumerate(tqdm(train_dataloader)):
        batch = {k: v.to(device) for k, v in batch.items()}
        outputs = model(**batch)
        loss = outputs.loss
        total_loss += loss.detach().float()
        loss.backward()
        optimizer.step()
        lr_scheduler.step()
        optimizer.zero_grad()

    model.eval()
    eval_loss = 0
    eval_preds = []
    for step, batch in enumerate(tqdm(eval_dataloader)):
        batch = {k: v.to(device) for k, v in batch.items()}
        with torch.no_grad():
            outputs = model(**batch)
        loss = outputs.loss
        eval_loss += loss.detach().float()
        eval_preds.extend(
            tokenizer.batch_decode(torch.argmax(outputs.logits, -1).detach().cpu().numpy(), skip_special_tokens=True)
        )

    eval_epoch_loss = eval_loss / len(eval_dataloader)
    eval_ppl = torch.exp(eval_epoch_loss)
    train_epoch_loss = total_loss / len(train_dataloader)
    train_ppl = torch.exp(train_epoch_loss)
    print(f"{epoch=}: {train_ppl=} {train_epoch_loss=} {eval_ppl=} {eval_epoch_loss=}")
```

## Share your model

Once training is complete, you can upload your model to the Hub with the [`~transformers.PreTrainedModel.push_to_hub`] method. You'll need to login to your Hugging Face account first and enter your token when prompted.


key: _node_content
string_value:
{"id_": "41e713e2-0188-45b1-a2ec-df4c73d17ccf", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/prompt_based_methods", "title": "Prompt-based methods", "tokens": 3315, "retrieve_doc": true, "source": "peft"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5abc71a2-6dbd-4e86-8702-af415be1c042", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/prompt_based_methods", "title": "Prompt-based methods", "tokens": 3315, "retrieve_doc": true, "source": "peft"}, "hash": "a4974d540eeb655582d44b88d415425379b0d06f7d5e927999502cad9d07e2c7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d2405782-9a0b-42bd-9fd1-f10a3518f0cd", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/prompt_based_methods", "title": "Prompt-based methods", "tokens": 3315, "retrieve_doc": true, "source": "peft"}, "hash": "7ea13b08901d3bc5e583118cc69a4b3e89a2861a940ad7c4ad23c9700773289b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9621a59e-518c-44af-8f30-7192f76dd8b1", "node_type": "1", "metadata": {}, "hash": "be68b9bfe457f9a31ad311f79345e48d928e1ba6583b2d43c8704c9e02280990", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9821, "end_char_idx": 12970, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
from tqdm import tqdm

device = "cuda"
model = model.to(device)

for epoch in range(num_epochs):
    model.train()
    total_loss = 0
    for step, batch in enumerate(tqdm(train_dataloader)):
        batch = {k: v.to(device) for k, v in batch.items()}
        outputs = model(**batch)
        loss = outputs.loss
        total_loss += loss.detach().float()
        loss.backward()
        optimizer.step()
        lr_scheduler.step()
        optimizer.zero_grad()

    model.eval()
    eval_loss = 0
    eval_preds = []
    for step, batch in enumerate(tqdm(eval_dataloader)):
        batch = {k: v.to(device) for k, v in batch.items()}
        with torch.no_grad():
            outputs = model(**batch)
        loss = outputs.loss
        eval_loss += loss.detach().float()
        eval_preds.extend(
            tokenizer.batch_decode(torch.argmax(outputs.logits, -1).detach().cpu().numpy(), skip_special_tokens=True)
        )

    eval_epoch_loss = eval_loss / len(eval_dataloader)
    eval_ppl = torch.exp(eval_epoch_loss)
    train_epoch_loss = total_loss / len(train_dataloader)
    train_ppl = torch.exp(train_epoch_loss)
    print(f"{epoch=}: {train_ppl=} {train_epoch_loss=} {eval_ppl=} {eval_epoch_loss=}")
```

## Share your model

Once training is complete, you can upload your model to the Hub with the [`~transformers.PreTrainedModel.push_to_hub`] method. You'll need to login to your Hugging Face account first and enter your token when prompted.

```py
from huggingface_hub import notebook_login

account = <your-hf-account-name>
peft_model_id = f"{account}/bloomz-560-m-peft-method"
model.push_to_hub(peft_model_id)
```

If you check the model file size in the repository, you’ll see that it is a lot smaller than a full sized model!

<div class="flex flex-col justify-center">
  <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/peft/PEFT-hub-screenshot.png"/>
  <figcaption class="text-center">For example, the adapter weights for a opt-350m model stored on the Hub are only ~6MB compared to the full model size which can be ~700MB.</figcaption>
</div>

## Inference

Let's load the model for inference and test it out on a tweet!

```py
from peft import AutoPeftModelForCausalLM

model = AutoPeftModelForCausalLM.from_pretrained("peft_model_id").to("cuda")
tokenizer = AutoTokenizer.from_pretrained("bigscience/bloomz-560m")

i = 15
inputs = tokenizer(f'{text_column} : {ds["test"][i]["Tweet text"]} Label : ', return_tensors="pt")
print(ds["test"][i]["Tweet text"])
"@NYTsupport i have complained a dozen times &amp; yet my papers are still thrown FAR from my door. Why is this so hard to resolve?"
```

Call the [`~transformers.GenerationMixin.generate`] method to generate the predicted classification label.

```py
with torch.no_grad():
    inputs = {k: v.to(device) for k, v in inputs.items()}
    outputs = model.generate(input_ids=inputs["input_ids"], max_new_tokens=10)
    print(tokenizer.batch_decode(outputs.detach().cpu().numpy(), skip_special_tokens=True))
"['Tweet text : @NYTsupport i have complained a dozen times &amp; yet my papers are still thrown FAR from my door.


key: _node_content
string_value:
{"id_": "9621a59e-518c-44af-8f30-7192f76dd8b1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/prompt_based_methods", "title": "Prompt-based methods", "tokens": 3315, "retrieve_doc": true, "source": "peft"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5abc71a2-6dbd-4e86-8702-af415be1c042", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/prompt_based_methods", "title": "Prompt-based methods", "tokens": 3315, "retrieve_doc": true, "source": "peft"}, "hash": "a4974d540eeb655582d44b88d415425379b0d06f7d5e927999502cad9d07e2c7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "41e713e2-0188-45b1-a2ec-df4c73d17ccf", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/peft/task_guides/prompt_based_methods", "title": "Prompt-based methods", "tokens": 3315, "retrieve_doc": true, "source": "peft"}, "hash": "0880326e5e0a2477e8b33dc9ff1cffd922b5cb1df84a36b18a5ce1ce7406bd96", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11580, "end_char_idx": 13027, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<div class="flex flex-col justify-center">
  <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/peft/PEFT-hub-screenshot.png"/>
  <figcaption class="text-center">For example, the adapter weights for a opt-350m model stored on the Hub are only ~6MB compared to the full model size which can be ~700MB.</figcaption>
</div>

## Inference

Let's load the model for inference and test it out on a tweet!

```py
from peft import AutoPeftModelForCausalLM

model = AutoPeftModelForCausalLM.from_pretrained("peft_model_id").to("cuda")
tokenizer = AutoTokenizer.from_pretrained("bigscience/bloomz-560m")

i = 15
inputs = tokenizer(f'{text_column} : {ds["test"][i]["Tweet text"]} Label : ', return_tensors="pt")
print(ds["test"][i]["Tweet text"])
"@NYTsupport i have complained a dozen times &amp; yet my papers are still thrown FAR from my door. Why is this so hard to resolve?"
```

Call the [`~transformers.GenerationMixin.generate`] method to generate the predicted classification label.

```py
with torch.no_grad():
    inputs = {k: v.to(device) for k, v in inputs.items()}
    outputs = model.generate(input_ids=inputs["input_ids"], max_new_tokens=10)
    print(tokenizer.batch_decode(outputs.detach().cpu().numpy(), skip_special_tokens=True))
"['Tweet text : @NYTsupport i have complained a dozen times &amp; yet my papers are still thrown FAR from my door. Why is this so hard to resolve? Label : complaint']"
```


