# embeddings_queue

created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: 13a87b1d-4d27-42a5-8afa-05f421326f70
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/developer_guides/lora", "title": "LoRA", "tokens": 4998, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"13a87b1d-4d27-42a5-8afa-05f421326f70\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/lora\", \"title\": \"LoRA\", \"tokens\": 4998, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1c57a8ee-656c-4c3c-bef0-2c2fc7823e4e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/lora\", \"title\": \"LoRA\", \"tokens\": 4998, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"b0064fa4dcc01d18535d58e704f09d04ac4a343d0268b28e4e4449bf362153d7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"977237cb-de61-4243-8e62-0f7ef6a88813\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/lora\", \"title\": \"LoRA\", \"tokens\": 4998, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"20a3d217b820e31db4c4b4fbded8e3c394af3a41ac0fd7d2d6bd542a47099121\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"dede7561-7373-469b-ac73-e3f08856b21c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"150098712c4e3305dfc6296f59ef0c6617f7468c635c763f4f91d106be7369df\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12950, \"end_char_idx\": 15940, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1c57a8ee-656c-4c3c-bef0-2c2fc7823e4e", "doc_id": "1c57a8ee-656c-4c3c-bef0-2c2fc7823e4e", "ref_doc_id": "1c57a8ee-656c-4c3c-bef0-2c2fc7823e4e", "chroma:document": "Below is an end-to-end example.\n\nFirst load the base model:\n\n```python\nfrom transformers import AutoModelForCausalLM\nfrom peft import PeftModel\nimport torch\n\nbase_model = AutoModelForCausalLM.from_pretrained(\n    \"mistralai/Mistral-7B-v0.1\", torch_dtype=torch.float16, device_map=\"auto\"\n)\n```\n\nThen we load the first adapter: \n\n```python\npeft_model_id = \"alignment-handbook/zephyr-7b-sft-lora\"\nmodel = PeftModel.from_pretrained(base_model, peft_model_id, adapter_name=\"sft\")\n```\n\nThen load a different adapter and merge it with the first one:\n\n```python\nweighted_adapter_name = \"sft-dpo\"\nmodel.load_adapter(\"alignment-handbook/zephyr-7b-dpo-lora\", adapter_name=\"dpo\")\nmodel.add_weighted_adapter(\n    adapters=[\"sft\", \"dpo\"],\n    weights=[0.7, 0.3],\n    adapter_name=weighted_adapter_name,\n    combination_type=\"linear\"\n)\nmodel.set_adapter(weighted_adapter_name)\n```\n\n<Tip>\n\nThere are several supported methods for `combination_type`. Refer to the [documentation](../package_reference/lora#peft.LoraModel.add_weighted_adapter) for more details. Note that \"svd\" as the `combination_type` is not supported when using `torch.float16` or `torch.bfloat16` as the datatype.\n\n</Tip>\n\nNow, perform inference:\n\n```python\ntokenizer = AutoTokenizer.from_pretrained(\"mistralai/Mistral-7B-v0.1\")\n\nprompt = \"Hey, are you conscious? Can you talk to me?\"\ninputs = tokenizer(prompt, return_tensors=\"pt\")\ninputs = {k: v.to(\"cuda\") for k, v in inputs.items()}\n\nwith torch.no_grad():\n    generate_ids = model.generate(**inputs, max_length=30)\noutputs = tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]\nprint(outputs)\n```\n\n## Load adapters\n\nAdapters can be loaded onto a pretrained model with [`~PeftModel.load_adapter`], which is useful for trying out different adapters whose weights aren't merged. Set the active adapter weights with the [`~LoraModel.set_adapter`] function.\n\n```py\nfrom transformers import AutoModelForCausalLM\nfrom peft import PeftModel\n\nbase_model = AutoModelForCausalLM.from_pretrained(\"mistralai/Mistral-7B-v0.1\")\npeft_model_id = \"alignment-handbook/zephyr-7b-sft-lora\"\nmodel = PeftModel.from_pretrained(base_model, peft_model_id)\n\n# load different adapter\nmodel.load_adapter(\"alignment-handbook/zephyr-7b-dpo-lora\", adapter_name=\"dpo\")\n\n# set adapter as active\nmodel.set_adapter(\"dpo\")\n```\n\nTo return the base model, you could use [`~LoraModel.unload`] to unload all of the LoRA modules or [`~LoraModel.delete_adapter`] to delete the adapter entirely.\n\n```py\n# unload adapter\nmodel.unload()\n\n# delete adapter\nmodel.delete_adapter(\"dpo\")\n```\n\n## Inference with different LoRA adapters in the same batch\n\nNormally, each inference batch has to use the same adapter(s) in PEFT. This can sometimes be annoying, because we may have batches that contain samples intended to be used with different LoRA adapters. For example, we could have a base model that works well in English and two more LoRA adapters, one for French and one for German."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: dede7561-7373-469b-ac73-e3f08856b21c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/developer_guides/lora", "title": "LoRA", "tokens": 4998, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"dede7561-7373-469b-ac73-e3f08856b21c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/lora\", \"title\": \"LoRA\", \"tokens\": 4998, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1c57a8ee-656c-4c3c-bef0-2c2fc7823e4e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/lora\", \"title\": \"LoRA\", \"tokens\": 4998, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"b0064fa4dcc01d18535d58e704f09d04ac4a343d0268b28e4e4449bf362153d7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"13a87b1d-4d27-42a5-8afa-05f421326f70\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/lora\", \"title\": \"LoRA\", \"tokens\": 4998, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"bd1de9c65b3d9c756800d506bdf999fe97713b98ad8570376cb89d240c70b8d0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"388abbdd-6b21-4f06-9709-237bb9b92218\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"62c4b0a1792699a95ebcdf268b7d3c74aa6cb293e8ab8958470ba8ab6ca1603a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14782, \"end_char_idx\": 17495, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1c57a8ee-656c-4c3c-bef0-2c2fc7823e4e", "doc_id": "1c57a8ee-656c-4c3c-bef0-2c2fc7823e4e", "ref_doc_id": "1c57a8ee-656c-4c3c-bef0-2c2fc7823e4e", "chroma:document": "Set the active adapter weights with the [`~LoraModel.set_adapter`] function.\n\n```py\nfrom transformers import AutoModelForCausalLM\nfrom peft import PeftModel\n\nbase_model = AutoModelForCausalLM.from_pretrained(\"mistralai/Mistral-7B-v0.1\")\npeft_model_id = \"alignment-handbook/zephyr-7b-sft-lora\"\nmodel = PeftModel.from_pretrained(base_model, peft_model_id)\n\n# load different adapter\nmodel.load_adapter(\"alignment-handbook/zephyr-7b-dpo-lora\", adapter_name=\"dpo\")\n\n# set adapter as active\nmodel.set_adapter(\"dpo\")\n```\n\nTo return the base model, you could use [`~LoraModel.unload`] to unload all of the LoRA modules or [`~LoraModel.delete_adapter`] to delete the adapter entirely.\n\n```py\n# unload adapter\nmodel.unload()\n\n# delete adapter\nmodel.delete_adapter(\"dpo\")\n```\n\n## Inference with different LoRA adapters in the same batch\n\nNormally, each inference batch has to use the same adapter(s) in PEFT. This can sometimes be annoying, because we may have batches that contain samples intended to be used with different LoRA adapters. For example, we could have a base model that works well in English and two more LoRA adapters, one for French and one for German. Usually, we would have to split our batches such that each batch only contains samples of one of the languages, we cannot combine different languages in the same batch.\n\nThankfully, it is possible to mix different LoRA adapters in the same batch using the `adapter_name` argument. Below, we show an example of how this works in practice. First, let's load the base model, English, and the two adapters, French and German, like this:\n\n```python\nfrom transformers import AutoTokenizer, AutoModelForCausalLM\nfrom peft import PeftModel\n\nmodel_id = ...\ntokenizer = AutoTokenizer.from_pretrained(model_id)\n\nmodel = AutoModelForCausalLM.from_pretrained(model_id)\n# load the LoRA adapter for French\npeft_model = PeftModel.from_pretrained(model, <path>, adapter_name=\"adapter_fr\")\n# next, load the LoRA adapter for German\npeft_model.load_adapter(<path>, adapter_name=\"adapter_de\")\n```\n\nNow, we want to generate text on a sample that contains all three languages: The first three samples are in English, the next three are in French, and the last three are in German. We can use the `adapter_names` argument to specify which adapter to use for each sample. Since our base model is used for English, we use the special string `\"__base__\"` for these samples. For the next three samples, we indicate the adapter name of the French LoRA fine-tune, in this case `\"adapter_fr\"`. For the last three samples, we indicate the adapter name of the German LoRA fine-tune, in this case `\"adapter_de\"`. This way, we can use the base model and the two adapters in a single batch."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: 388abbdd-6b21-4f06-9709-237bb9b92218
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/developer_guides/lora", "title": "LoRA", "tokens": 4998, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"388abbdd-6b21-4f06-9709-237bb9b92218\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/lora\", \"title\": \"LoRA\", \"tokens\": 4998, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1c57a8ee-656c-4c3c-bef0-2c2fc7823e4e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/lora\", \"title\": \"LoRA\", \"tokens\": 4998, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"b0064fa4dcc01d18535d58e704f09d04ac4a343d0268b28e4e4449bf362153d7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"dede7561-7373-469b-ac73-e3f08856b21c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/lora\", \"title\": \"LoRA\", \"tokens\": 4998, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"0c163d7c4e550d437c82a8989581e4e2ce462ecceb59be5814002c9a7cb5b14a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"23dae7bc-bea8-4ba2-8853-b280e824b9b4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c2e917841febca1dac9ffac308eb61271d2349e174dfdb77a5032463e8ba5ddb\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 15811, \"end_char_idx\": 19090, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1c57a8ee-656c-4c3c-bef0-2c2fc7823e4e", "doc_id": "1c57a8ee-656c-4c3c-bef0-2c2fc7823e4e", "ref_doc_id": "1c57a8ee-656c-4c3c-bef0-2c2fc7823e4e", "chroma:document": "For example, we could have a base model that works well in English and two more LoRA adapters, one for French and one for German. Usually, we would have to split our batches such that each batch only contains samples of one of the languages, we cannot combine different languages in the same batch.\n\nThankfully, it is possible to mix different LoRA adapters in the same batch using the `adapter_name` argument. Below, we show an example of how this works in practice. First, let's load the base model, English, and the two adapters, French and German, like this:\n\n```python\nfrom transformers import AutoTokenizer, AutoModelForCausalLM\nfrom peft import PeftModel\n\nmodel_id = ...\ntokenizer = AutoTokenizer.from_pretrained(model_id)\n\nmodel = AutoModelForCausalLM.from_pretrained(model_id)\n# load the LoRA adapter for French\npeft_model = PeftModel.from_pretrained(model, <path>, adapter_name=\"adapter_fr\")\n# next, load the LoRA adapter for German\npeft_model.load_adapter(<path>, adapter_name=\"adapter_de\")\n```\n\nNow, we want to generate text on a sample that contains all three languages: The first three samples are in English, the next three are in French, and the last three are in German. We can use the `adapter_names` argument to specify which adapter to use for each sample. Since our base model is used for English, we use the special string `\"__base__\"` for these samples. For the next three samples, we indicate the adapter name of the French LoRA fine-tune, in this case `\"adapter_fr\"`. For the last three samples, we indicate the adapter name of the German LoRA fine-tune, in this case `\"adapter_de\"`. This way, we can use the base model and the two adapters in a single batch.\n\n```python\ninputs = tokenizer(\n    [\n        \"Hello, my dog is cute\",\n        \"Hello, my cat is awesome\",\n        \"Hello, my fish is great\",\n        \"Salut, mon chien est mignon\",\n        \"Salut, mon chat est g\u00e9nial\",\n        \"Salut, mon poisson est super\",\n        \"Hallo, mein Hund ist s\u00fc\u00df\",\n        \"Hallo, meine Katze ist toll\",\n        \"Hallo, mein Fisch ist gro\u00dfartig\",\n    ],\n    return_tensors=\"pt\",\n    padding=True,\n)\n\nadapter_names = [\n    \"__base__\", \"__base__\", \"__base__\",\n    \"adapter_fr\", \"adapter_fr\", \"adapter_fr\",\n    \"adapter_de\", \"adapter_de\", \"adapter_de\",\n]\noutput = peft_model.generate(**inputs, adapter_names=adapter_names, max_new_tokens=20)\n```\n\nNote that the order does not matter here, i.e. the samples in the batch don't need to be grouped by adapter as in the example above. We just need to ensure that the `adapter_names` argument is aligned correctly with the samples.\n\n### Caveats\n\nUsing this features has some drawbacks, namely:\n\n- It only works for inference, not for training.\n- Disabling adapters using the `with model.disable_adapter()` context takes precedence over `adapter_names`.\n- You cannot pass `adapter_names` when some adapter weights where merged with base weight using the `merge_adapter` method. Please unmerge all adapters first by calling `model.unmerge_adapter()`.\n- For obvious reasons, this cannot be used after calling `merge_and_unload()`, since all the LoRA adapters will be merged into the base weights in this case.\n- This feature does not currently work with DoRA, so set `use_dora=False` in your `LoraConfig` if you want to use it."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: 23dae7bc-bea8-4ba2-8853-b280e824b9b4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/developer_guides/lora", "title": "LoRA", "tokens": 4998, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"23dae7bc-bea8-4ba2-8853-b280e824b9b4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/lora\", \"title\": \"LoRA\", \"tokens\": 4998, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1c57a8ee-656c-4c3c-bef0-2c2fc7823e4e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/lora\", \"title\": \"LoRA\", \"tokens\": 4998, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"b0064fa4dcc01d18535d58e704f09d04ac4a343d0268b28e4e4449bf362153d7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"388abbdd-6b21-4f06-9709-237bb9b92218\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/lora\", \"title\": \"LoRA\", \"tokens\": 4998, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"89e45f291a9ba5023b785bc2c8632721cb23f87e34aa3b38c20313b945e4d9cf\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 17497, \"end_char_idx\": 19923, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1c57a8ee-656c-4c3c-bef0-2c2fc7823e4e", "doc_id": "1c57a8ee-656c-4c3c-bef0-2c2fc7823e4e", "ref_doc_id": "1c57a8ee-656c-4c3c-bef0-2c2fc7823e4e", "chroma:document": "```python\ninputs = tokenizer(\n    [\n        \"Hello, my dog is cute\",\n        \"Hello, my cat is awesome\",\n        \"Hello, my fish is great\",\n        \"Salut, mon chien est mignon\",\n        \"Salut, mon chat est g\u00e9nial\",\n        \"Salut, mon poisson est super\",\n        \"Hallo, mein Hund ist s\u00fc\u00df\",\n        \"Hallo, meine Katze ist toll\",\n        \"Hallo, mein Fisch ist gro\u00dfartig\",\n    ],\n    return_tensors=\"pt\",\n    padding=True,\n)\n\nadapter_names = [\n    \"__base__\", \"__base__\", \"__base__\",\n    \"adapter_fr\", \"adapter_fr\", \"adapter_fr\",\n    \"adapter_de\", \"adapter_de\", \"adapter_de\",\n]\noutput = peft_model.generate(**inputs, adapter_names=adapter_names, max_new_tokens=20)\n```\n\nNote that the order does not matter here, i.e. the samples in the batch don't need to be grouped by adapter as in the example above. We just need to ensure that the `adapter_names` argument is aligned correctly with the samples.\n\n### Caveats\n\nUsing this features has some drawbacks, namely:\n\n- It only works for inference, not for training.\n- Disabling adapters using the `with model.disable_adapter()` context takes precedence over `adapter_names`.\n- You cannot pass `adapter_names` when some adapter weights where merged with base weight using the `merge_adapter` method. Please unmerge all adapters first by calling `model.unmerge_adapter()`.\n- For obvious reasons, this cannot be used after calling `merge_and_unload()`, since all the LoRA adapters will be merged into the base weights in this case.\n- This feature does not currently work with DoRA, so set `use_dora=False` in your `LoraConfig` if you want to use it.\n- There is an expected overhead for inference with `adapter_names`, especially if the amount of different adapters in the batch is high. This is because the batch size is effectively reduced to the number of samples per adapter. If runtime performance is your top priority, try the following:\n  - Increase the batch size.\n  - Try to avoid having a large number of different adapters in the same batch, prefer homogeneous batches. This can be achieved by buffering samples with the same adapter and only perform inference with a small handfull of different adapters.\n  - Take a look at alternative implementations such as [LoRAX](https://github.com/predibase/lorax), [punica](https://github.com/punica-ai/punica), or [S-LoRA](https://github.com/S-LoRA/S-LoRA), which are specialized to work with a large number of different adapters."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: 2565e59b-8bde-4a98-ab67-a5f47518d094
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/developer_guides/custom_models", "title": "Custom models", "tokens": 3842, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"2565e59b-8bde-4a98-ab67-a5f47518d094\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/custom_models\", \"title\": \"Custom models\", \"tokens\": 3842, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"587b2ffc-a94b-4f12-a621-18f994f432f3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/custom_models\", \"title\": \"Custom models\", \"tokens\": 3842, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"f07eb5625e8d53aa533339337f7a581384cdf5d991cc351a3ce6956fbd60c213\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"12288c14-cf80-4e6d-8b97-698b129856a3\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5c487d37a39a95b6f191d18a1c4b9ba33eca99e99aa8f4081b29ab55137ffae1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3097, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "587b2ffc-a94b-4f12-a621-18f994f432f3", "doc_id": "587b2ffc-a94b-4f12-a621-18f994f432f3", "ref_doc_id": "587b2ffc-a94b-4f12-a621-18f994f432f3", "chroma:document": "# Custom models\n\nSome fine-tuning techniques, such as prompt tuning, are specific to language models. That means in \ud83e\udd17 PEFT, it is\nassumed a \ud83e\udd17 Transformers model is being used. However, other fine-tuning techniques - like\n[LoRA](../conceptual_guides/lora) - are not restricted to specific model types.\n\nIn this guide, we will see how LoRA can be applied to a multilayer perceptron, a computer vision model from the [timm](https://huggingface.co/docs/timm/index) library, or a new \ud83e\udd17 Transformers architecture.\n\n## Multilayer perceptron\n\nLet's assume that we want to fine-tune a multilayer perceptron with LoRA. Here is the definition:\n\n```python\nfrom torch import nn\n\n\nclass MLP(nn.Module):\n    def __init__(self, num_units_hidden=2000):\n        super().__init__()\n        self.seq = nn.Sequential(\n            nn.Linear(20, num_units_hidden),\n            nn.ReLU(),\n            nn.Linear(num_units_hidden, num_units_hidden),\n            nn.ReLU(),\n            nn.Linear(num_units_hidden, 2),\n            nn.LogSoftmax(dim=-1),\n        )\n\n    def forward(self, X):\n        return self.seq(X)\n```\n\nThis is a straightforward multilayer perceptron with an input layer, a hidden layer, and an output layer.\n\n<Tip>\n\nFor this toy example, we choose an exceedingly large number of hidden units to highlight the efficiency gains\nfrom PEFT, but those gains are in line with more realistic examples.\n\n</Tip>\n\nThere are a few linear layers in this model that could be tuned with LoRA. When working with common \ud83e\udd17 Transformers\nmodels, PEFT will know which layers to apply LoRA to, but in this case, it is up to us as a user to choose the layers.\nTo determine the names of the layers to tune:\n\n```python\nprint([(n, type(m)) for n, m in MLP().named_modules()])\n```\n\nThis should print:\n\n```\n[('', __main__.MLP),\n ('seq', torch.nn.modules.container.Sequential),\n ('seq.0', torch.nn.modules.linear.Linear),\n ('seq.1', torch.nn.modules.activation.ReLU),\n ('seq.2', torch.nn.modules.linear.Linear),\n ('seq.3', torch.nn.modules.activation.ReLU),\n ('seq.4', torch.nn.modules.linear.Linear),\n ('seq.5', torch.nn.modules.activation.LogSoftmax)]\n```\n\nLet's say we want to apply LoRA to the input layer and to the hidden layer, those are `'seq.0'` and `'seq.2'`. Moreover,\nlet's assume we want to update the output layer without LoRA, that would be `'seq.4'`. The corresponding config would\nbe:\n\n```python\nfrom peft import LoraConfig\n\nconfig = LoraConfig(\n    target_modules=[\"seq.0\", \"seq.2\"],\n    modules_to_save=[\"seq.4\"],\n)\n```\n\nWith that, we can create our PEFT model and check the fraction of parameters trained:\n\n```python\nfrom peft import get_peft_model\n\nmodel = MLP()\npeft_model = get_peft_model(model, config)\npeft_model.print_trainable_parameters()\n# prints trainable params: 56,164 || all params: 4,100,164 || trainable%: 1.369798866581922\n```\n\nFinally, we can use any training framework we like, or write our own fit loop, to train the `peft_model`.\n\nFor a complete example, check out [this notebook](https://github.com/huggingface/peft/blob/main/examples/multilayer_perceptron/multilayer_perceptron_lora.ipynb)."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: 12288c14-cf80-4e6d-8b97-698b129856a3
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/developer_guides/custom_models", "title": "Custom models", "tokens": 3842, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"12288c14-cf80-4e6d-8b97-698b129856a3\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/custom_models\", \"title\": \"Custom models\", \"tokens\": 3842, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"587b2ffc-a94b-4f12-a621-18f994f432f3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/custom_models\", \"title\": \"Custom models\", \"tokens\": 3842, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"f07eb5625e8d53aa533339337f7a581384cdf5d991cc351a3ce6956fbd60c213\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2565e59b-8bde-4a98-ab67-a5f47518d094\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/custom_models\", \"title\": \"Custom models\", \"tokens\": 3842, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"8e46a1662cd9344d6fe3f4ff1bbae191dc15390f9b229518e092b08b1e6f764c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"83d29605-18ce-4103-bf48-f7873c9baf16\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5b2973a013ec1ae6e35a9b894d0a096fc1bfb5562f746d8662ebb1c7734be15e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1631, \"end_char_idx\": 3867, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "587b2ffc-a94b-4f12-a621-18f994f432f3", "doc_id": "587b2ffc-a94b-4f12-a621-18f994f432f3", "ref_doc_id": "587b2ffc-a94b-4f12-a621-18f994f432f3", "chroma:document": "To determine the names of the layers to tune:\n\n```python\nprint([(n, type(m)) for n, m in MLP().named_modules()])\n```\n\nThis should print:\n\n```\n[('', __main__.MLP),\n ('seq', torch.nn.modules.container.Sequential),\n ('seq.0', torch.nn.modules.linear.Linear),\n ('seq.1', torch.nn.modules.activation.ReLU),\n ('seq.2', torch.nn.modules.linear.Linear),\n ('seq.3', torch.nn.modules.activation.ReLU),\n ('seq.4', torch.nn.modules.linear.Linear),\n ('seq.5', torch.nn.modules.activation.LogSoftmax)]\n```\n\nLet's say we want to apply LoRA to the input layer and to the hidden layer, those are `'seq.0'` and `'seq.2'`. Moreover,\nlet's assume we want to update the output layer without LoRA, that would be `'seq.4'`. The corresponding config would\nbe:\n\n```python\nfrom peft import LoraConfig\n\nconfig = LoraConfig(\n    target_modules=[\"seq.0\", \"seq.2\"],\n    modules_to_save=[\"seq.4\"],\n)\n```\n\nWith that, we can create our PEFT model and check the fraction of parameters trained:\n\n```python\nfrom peft import get_peft_model\n\nmodel = MLP()\npeft_model = get_peft_model(model, config)\npeft_model.print_trainable_parameters()\n# prints trainable params: 56,164 || all params: 4,100,164 || trainable%: 1.369798866581922\n```\n\nFinally, we can use any training framework we like, or write our own fit loop, to train the `peft_model`.\n\nFor a complete example, check out [this notebook](https://github.com/huggingface/peft/blob/main/examples/multilayer_perceptron/multilayer_perceptron_lora.ipynb).\n\n## timm models\n\nThe [timm](https://huggingface.co/docs/timm/index) library contains a large number of pretrained computer vision models.\nThose can also be fine-tuned with PEFT. Let's check out how this works in practice.\n\nTo start, ensure that timm is installed in the Python environment:\n\n```bash\npython -m pip install -U timm\n```\n\nNext we load a timm model for an image classification task:\n\n```python\nimport timm\n\nnum_classes = ...\nmodel_id = \"timm/poolformer_m36.sail_in1k\"\nmodel = timm.create_model(model_id, pretrained=True, num_classes=num_classes)\n```\n\nAgain, we need to make a decision about what layers to apply LoRA to. Since LoRA supports 2D conv layers, and since\nthose are a major building block of this model, we should apply LoRA to the 2D conv layers."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: 83d29605-18ce-4103-bf48-f7873c9baf16
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/developer_guides/custom_models", "title": "Custom models", "tokens": 3842, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"83d29605-18ce-4103-bf48-f7873c9baf16\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/custom_models\", \"title\": \"Custom models\", \"tokens\": 3842, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"587b2ffc-a94b-4f12-a621-18f994f432f3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/custom_models\", \"title\": \"Custom models\", \"tokens\": 3842, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"f07eb5625e8d53aa533339337f7a581384cdf5d991cc351a3ce6956fbd60c213\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"12288c14-cf80-4e6d-8b97-698b129856a3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/custom_models\", \"title\": \"Custom models\", \"tokens\": 3842, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"9faaa7111d8bbd92fabe8d86e84cddf7a830005a6f3fff9613636ea296b04814\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"923ec8b7-95f2-479f-9c28-c574bf94c86c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a3d640b1a4850e33d4799b8727f1a7ad3edbd82f47a28f6b8208154464cd235f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2936, \"end_char_idx\": 6348, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "587b2ffc-a94b-4f12-a621-18f994f432f3", "doc_id": "587b2ffc-a94b-4f12-a621-18f994f432f3", "ref_doc_id": "587b2ffc-a94b-4f12-a621-18f994f432f3", "chroma:document": "For a complete example, check out [this notebook](https://github.com/huggingface/peft/blob/main/examples/multilayer_perceptron/multilayer_perceptron_lora.ipynb).\n\n## timm models\n\nThe [timm](https://huggingface.co/docs/timm/index) library contains a large number of pretrained computer vision models.\nThose can also be fine-tuned with PEFT. Let's check out how this works in practice.\n\nTo start, ensure that timm is installed in the Python environment:\n\n```bash\npython -m pip install -U timm\n```\n\nNext we load a timm model for an image classification task:\n\n```python\nimport timm\n\nnum_classes = ...\nmodel_id = \"timm/poolformer_m36.sail_in1k\"\nmodel = timm.create_model(model_id, pretrained=True, num_classes=num_classes)\n```\n\nAgain, we need to make a decision about what layers to apply LoRA to. Since LoRA supports 2D conv layers, and since\nthose are a major building block of this model, we should apply LoRA to the 2D conv layers. To identify the names of\nthose layers, let's look at all the layer names:\n\n```python\nprint([(n, type(m)) for n, m in model.named_modules()])\n```\n\nThis will print a very long list, we'll only show the first few:\n\n```\n[('', timm.models.metaformer.MetaFormer),\n ('stem', timm.models.metaformer.Stem),\n ('stem.conv', torch.nn.modules.conv.Conv2d),\n ('stem.norm', torch.nn.modules.linear.Identity),\n ('stages', torch.nn.modules.container.Sequential),\n ('stages.0', timm.models.metaformer.MetaFormerStage),\n ('stages.0.downsample', torch.nn.modules.linear.Identity),\n ('stages.0.blocks', torch.nn.modules.container.Sequential),\n ('stages.0.blocks.0', timm.models.metaformer.MetaFormerBlock),\n ('stages.0.blocks.0.norm1', timm.layers.norm.GroupNorm1),\n ('stages.0.blocks.0.token_mixer', timm.models.metaformer.Pooling),\n ('stages.0.blocks.0.token_mixer.pool', torch.nn.modules.pooling.AvgPool2d),\n ('stages.0.blocks.0.drop_path1', torch.nn.modules.linear.Identity),\n ('stages.0.blocks.0.layer_scale1', timm.models.metaformer.Scale),\n ('stages.0.blocks.0.res_scale1', torch.nn.modules.linear.Identity),\n ('stages.0.blocks.0.norm2', timm.layers.norm.GroupNorm1),\n ('stages.0.blocks.0.mlp', timm.layers.mlp.Mlp),\n ('stages.0.blocks.0.mlp.fc1', torch.nn.modules.conv.Conv2d),\n ('stages.0.blocks.0.mlp.act', torch.nn.modules.activation.GELU),\n ('stages.0.blocks.0.mlp.drop1', torch.nn.modules.dropout.Dropout),\n ('stages.0.blocks.0.mlp.norm', torch.nn.modules.linear.Identity),\n ('stages.0.blocks.0.mlp.fc2', torch.nn.modules.conv.Conv2d),\n ('stages.0.blocks.0.mlp.drop2', torch.nn.modules.dropout.Dropout),\n ('stages.0.blocks.0.drop_path2', torch.nn.modules.linear.Identity),\n ('stages.0.blocks.0.layer_scale2', timm.models.metaformer.Scale),\n ('stages.0.blocks.0.res_scale2', torch.nn.modules.linear.Identity),\n ('stages.0.blocks.1', timm.models.metaformer.MetaFormerBlock),\n ('stages.0.blocks.1.norm1', timm.layers.norm.GroupNorm1),\n ('stages.0.blocks.1.token_mixer', timm.models.metaformer.Pooling),\n ('stages.0.blocks.1.token_mixer.pool', torch.nn.modules.pooling.AvgPool2d),\n ...\n ('head.global_pool.flatten', torch.nn.modules.linear.Identity),\n ('head.norm', timm.layers.norm.LayerNorm2d),\n ('head.flatten', torch.nn.modules.flatten.Flatten),\n ('head.drop', torch.nn.modules.linear.Identity),\n ('head.fc', torch.nn.modules.linear.Linear)]\n ]\n```\n\nUpon closer inspection, we see that the 2D conv layers have names such as `\"stages.0.blocks.0.mlp.fc1\"` and\n`\"stages.0.blocks.0.mlp.fc2\"`."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: 923ec8b7-95f2-479f-9c28-c574bf94c86c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/developer_guides/custom_models", "title": "Custom models", "tokens": 3842, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"923ec8b7-95f2-479f-9c28-c574bf94c86c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/custom_models\", \"title\": \"Custom models\", \"tokens\": 3842, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"587b2ffc-a94b-4f12-a621-18f994f432f3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/custom_models\", \"title\": \"Custom models\", \"tokens\": 3842, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"f07eb5625e8d53aa533339337f7a581384cdf5d991cc351a3ce6956fbd60c213\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"83d29605-18ce-4103-bf48-f7873c9baf16\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/custom_models\", \"title\": \"Custom models\", \"tokens\": 3842, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"e43e8ecfdf56ee9c24cfe137585e4f1911c6366f72324a726e9fdf1e8ad72cf0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e71229e0-d9d8-4020-ae5d-5d186b006244\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0d0c92e82d7092acac39fee2d0f7cb4976bfcdd3b21817f9178f08f8974a177b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6349, \"end_char_idx\": 9414, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "587b2ffc-a94b-4f12-a621-18f994f432f3", "doc_id": "587b2ffc-a94b-4f12-a621-18f994f432f3", "ref_doc_id": "587b2ffc-a94b-4f12-a621-18f994f432f3", "chroma:document": "How can we match those layer names specifically? You can write a [regular\nexpressions](https://docs.python.org/3/library/re.html) to match the layer names. For our case, the regex\n`r\".*\\.mlp\\.fc\\d\"` should do the job.\n\nFurthermore, as in the first example, we should ensure that the output layer, in this case the classification head, is\nalso updated. Looking at the end of the list printed above, we can see that it's named `'head.fc'`. With that in mind,\nhere is our LoRA config:\n\n```python\nconfig = LoraConfig(target_modules=r\".*\\.mlp\\.fc\\d\", modules_to_save=[\"head.fc\"])\n```\n\nThen we only need to create the PEFT model by passing our base model and the config to `get_peft_model`:\n\n```python\npeft_model = get_peft_model(model, config)\npeft_model.print_trainable_parameters()\n# prints trainable params: 1,064,454 || all params: 56,467,974 || trainable%: 1.88505789139876\n```\n\nThis shows us that we only need to train less than 2% of all parameters, which is a huge efficiency gain.\n\nFor a complete example, check out [this notebook](https://github.com/huggingface/peft/blob/main/examples/image_classification/image_classification_timm_peft_lora.ipynb).\n\n## New transformers architectures\n\nWhen new popular transformers architectures are released, we do our best to quickly add them to PEFT. If you come across a transformers model that is not supported out of the box, don't worry, it will most likely still work if the config is set correctly. Specifically, you have to identify the layers that should be adapted and set them correctly when initializing the corresponding config class, e.g. `LoraConfig`. Here are some tips to help with this.\n\nAs a first step, it is a good idea is to check the existing models for inspiration. You can find them inside of [constants.py](https://github.com/huggingface/peft/blob/main/src/peft/utils/constants.py) in the PEFT repository. Often, you'll find a similar architecture that uses the same names. For example, if the new model architecture is a variation of the \"mistral\" model and you want to apply LoRA, you can see that the entry for \"mistral\" in `TRANSFORMERS_MODELS_TO_LORA_TARGET_MODULES_MAPPING` contains `[\"q_proj\", \"v_proj\"]`. This tells you that for \"mistral\" models, the `target_modules` for LoRA should be `[\"q_proj\", \"v_proj\"]`:\n\n```python\nfrom peft import LoraConfig, get_peft_model\n\nmy_mistral_model = ...\nconfig = LoraConfig(\n    target_modules=[\"q_proj\", \"v_proj\"],\n    ...,  # other LoRA arguments\n)\npeft_model = get_peft_model(my_mistral_model, config)\n```\n\nIf that doesn't help, check the existing modules in your model architecture with the `named_modules` method and try to identify the attention layers, especially the key, query, and value layers. Those will often have names such as `c_attn`, `query`, `q_proj`, etc. The key layer is not always adapted, and ideally, you should check whether including it results in better performance.\n\nAdditionally, linear layers are common targets to be adapted (e.g. in [QLoRA paper](https://arxiv.org/abs/2305.14314), authors suggest to adapt them as well)."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: e71229e0-d9d8-4020-ae5d-5d186b006244
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/developer_guides/custom_models", "title": "Custom models", "tokens": 3842, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"e71229e0-d9d8-4020-ae5d-5d186b006244\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/custom_models\", \"title\": \"Custom models\", \"tokens\": 3842, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"587b2ffc-a94b-4f12-a621-18f994f432f3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/custom_models\", \"title\": \"Custom models\", \"tokens\": 3842, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"f07eb5625e8d53aa533339337f7a581384cdf5d991cc351a3ce6956fbd60c213\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"923ec8b7-95f2-479f-9c28-c574bf94c86c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/custom_models\", \"title\": \"Custom models\", \"tokens\": 3842, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"f47dd33ceb84b93bb18414b48e779f6bc6da1ae39aa8d3b1e4672918c4c91028\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"60240726-63f3-4ae7-8246-fa917dc3724c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"71a9c2037532ef64cf30613285a620c1c2edec93f4c8be9af33a72cec46c9283\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7944, \"end_char_idx\": 10879, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "587b2ffc-a94b-4f12-a621-18f994f432f3", "doc_id": "587b2ffc-a94b-4f12-a621-18f994f432f3", "ref_doc_id": "587b2ffc-a94b-4f12-a621-18f994f432f3", "chroma:document": "`LoraConfig`. Here are some tips to help with this.\n\nAs a first step, it is a good idea is to check the existing models for inspiration. You can find them inside of [constants.py](https://github.com/huggingface/peft/blob/main/src/peft/utils/constants.py) in the PEFT repository. Often, you'll find a similar architecture that uses the same names. For example, if the new model architecture is a variation of the \"mistral\" model and you want to apply LoRA, you can see that the entry for \"mistral\" in `TRANSFORMERS_MODELS_TO_LORA_TARGET_MODULES_MAPPING` contains `[\"q_proj\", \"v_proj\"]`. This tells you that for \"mistral\" models, the `target_modules` for LoRA should be `[\"q_proj\", \"v_proj\"]`:\n\n```python\nfrom peft import LoraConfig, get_peft_model\n\nmy_mistral_model = ...\nconfig = LoraConfig(\n    target_modules=[\"q_proj\", \"v_proj\"],\n    ...,  # other LoRA arguments\n)\npeft_model = get_peft_model(my_mistral_model, config)\n```\n\nIf that doesn't help, check the existing modules in your model architecture with the `named_modules` method and try to identify the attention layers, especially the key, query, and value layers. Those will often have names such as `c_attn`, `query`, `q_proj`, etc. The key layer is not always adapted, and ideally, you should check whether including it results in better performance.\n\nAdditionally, linear layers are common targets to be adapted (e.g. in [QLoRA paper](https://arxiv.org/abs/2305.14314), authors suggest to adapt them as well). Their names will often contain the strings `fc` or `dense`.\n\nIf you want to add a new model to PEFT, please create an entry in [constants.py](https://github.com/huggingface/peft/blob/main/src/peft/utils/constants.py) and open a pull request on the [repository](https://github.com/huggingface/peft/pulls). Don't forget to update the [README](https://github.com/huggingface/peft#models-support-matrix) as well.\n\n## Verify parameters and layers\n\nYou can verify whether you've correctly applied a PEFT method to your model in a few ways.\n\n* Check the fraction of parameters that are trainable with the [`~PeftModel.print_trainable_parameters`] method. If this number is lower or higher than expected, check the model `repr` by printing the model. This shows the names of all the layer types in the model. Ensure that only the intended target layers are replaced by the adapter layers. For example, if LoRA is applied to `nn.Linear` layers, then you should only see `lora.Linear` layers being used.\n\n```py\npeft_model.print_trainable_parameters()\n```\n\n* Another way you can view the adapted layers is to use the `targeted_module_names` attribute to list the name of each module that was adapted.\n\n```python\nprint(peft_model.targeted_module_names)\n```\n\n## Unsupported module types\n\nMethods like LoRA only work if the target modules are supported by PEFT. For example, it's possible to apply LoRA to `nn.Linear` and `nn.Conv2d` layers, but not, for instance, to `nn.LSTM`."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: 60240726-63f3-4ae7-8246-fa917dc3724c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/developer_guides/custom_models", "title": "Custom models", "tokens": 3842, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"60240726-63f3-4ae7-8246-fa917dc3724c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/custom_models\", \"title\": \"Custom models\", \"tokens\": 3842, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"587b2ffc-a94b-4f12-a621-18f994f432f3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/custom_models\", \"title\": \"Custom models\", \"tokens\": 3842, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"f07eb5625e8d53aa533339337f7a581384cdf5d991cc351a3ce6956fbd60c213\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e71229e0-d9d8-4020-ae5d-5d186b006244\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/custom_models\", \"title\": \"Custom models\", \"tokens\": 3842, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"4e0d232c8776e1f0815bb8b2a936b67df671f8a21528beea7717ca817d717faa\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e582550d-a5e8-46a9-a10d-827048c1b4c8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"335ba5ff2a2d338c9281c5f4083251ad927b07c4181524ace9fd4c6a257a1aa7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9256, \"end_char_idx\": 12525, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "587b2ffc-a94b-4f12-a621-18f994f432f3", "doc_id": "587b2ffc-a94b-4f12-a621-18f994f432f3", "ref_doc_id": "587b2ffc-a94b-4f12-a621-18f994f432f3", "chroma:document": "Additionally, linear layers are common targets to be adapted (e.g. in [QLoRA paper](https://arxiv.org/abs/2305.14314), authors suggest to adapt them as well). Their names will often contain the strings `fc` or `dense`.\n\nIf you want to add a new model to PEFT, please create an entry in [constants.py](https://github.com/huggingface/peft/blob/main/src/peft/utils/constants.py) and open a pull request on the [repository](https://github.com/huggingface/peft/pulls). Don't forget to update the [README](https://github.com/huggingface/peft#models-support-matrix) as well.\n\n## Verify parameters and layers\n\nYou can verify whether you've correctly applied a PEFT method to your model in a few ways.\n\n* Check the fraction of parameters that are trainable with the [`~PeftModel.print_trainable_parameters`] method. If this number is lower or higher than expected, check the model `repr` by printing the model. This shows the names of all the layer types in the model. Ensure that only the intended target layers are replaced by the adapter layers. For example, if LoRA is applied to `nn.Linear` layers, then you should only see `lora.Linear` layers being used.\n\n```py\npeft_model.print_trainable_parameters()\n```\n\n* Another way you can view the adapted layers is to use the `targeted_module_names` attribute to list the name of each module that was adapted.\n\n```python\nprint(peft_model.targeted_module_names)\n```\n\n## Unsupported module types\n\nMethods like LoRA only work if the target modules are supported by PEFT. For example, it's possible to apply LoRA to `nn.Linear` and `nn.Conv2d` layers, but not, for instance, to `nn.LSTM`. If you find a layer class you want to apply PEFT to is not supported, you can:\n\n - define a custom mapping to dynamically dispatch custom modules in LoRA\n -  open an [issue](https://github.com/huggingface/peft/issues) and request the feature where maintainers will implement it or guide you on how to implement it yourself if demand for this module type is sufficiently high\n\n### Experimental support for dynamic dispatch of custom modules in LoRA\n\n> [!WARNING]\n> This feature is experimental and subject to change, depending on its reception by the community. We will introduce a public and stable API if there is significant demand for it.\n\nPEFT supports an experimental API for custom module types for LoRA. Let's assume you have a LoRA implementation for LSTMs. Normally, you would not be able to tell PEFT to use it, even if it would theoretically work with PEFT. However, this is possible with dynamic dispatch of custom layers.\n\nThe experimental API currently looks like this:\n\n```python\nclass MyLoraLSTMLayer:\n    ...\n\nbase_model = ...  # load the base model that uses LSTMs\n\n# add the LSTM layer names to target_modules\nconfig = LoraConfig(..., target_modules=[\"lstm\"])\n# define a mapping from base layer type to LoRA layer type\ncustom_module_mapping = {nn.LSTM: MyLoraLSTMLayer}\n# register the new mapping\nconfig._register_custom_module(custom_module_mapping)\n# after registration, create the PEFT model\npeft_model = get_peft_model(base_model, config)\n# do training\n```\n\n<Tip>\n\nWhen you call [`get_peft_model`], you will see a warning because PEFT does not recognize the targeted module type. In this case, you can ignore this warning."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: e582550d-a5e8-46a9-a10d-827048c1b4c8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/developer_guides/custom_models", "title": "Custom models", "tokens": 3842, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"e582550d-a5e8-46a9-a10d-827048c1b4c8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/custom_models\", \"title\": \"Custom models\", \"tokens\": 3842, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"587b2ffc-a94b-4f12-a621-18f994f432f3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/custom_models\", \"title\": \"Custom models\", \"tokens\": 3842, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"f07eb5625e8d53aa533339337f7a581384cdf5d991cc351a3ce6956fbd60c213\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"60240726-63f3-4ae7-8246-fa917dc3724c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/custom_models\", \"title\": \"Custom models\", \"tokens\": 3842, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"e4c36706bb8bf30757a0644a5b7a4f07e28fcbe6b5d054c2315803155be8c443\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"09c06682-3d86-4316-8d85-11cc07712a06\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"59f8141b8e00761f09fdbc25bef2ee022bcec178ebe378b99421dba8eb97343d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10880, \"end_char_idx\": 14083, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "587b2ffc-a94b-4f12-a621-18f994f432f3", "doc_id": "587b2ffc-a94b-4f12-a621-18f994f432f3", "ref_doc_id": "587b2ffc-a94b-4f12-a621-18f994f432f3", "chroma:document": "If you find a layer class you want to apply PEFT to is not supported, you can:\n\n - define a custom mapping to dynamically dispatch custom modules in LoRA\n -  open an [issue](https://github.com/huggingface/peft/issues) and request the feature where maintainers will implement it or guide you on how to implement it yourself if demand for this module type is sufficiently high\n\n### Experimental support for dynamic dispatch of custom modules in LoRA\n\n> [!WARNING]\n> This feature is experimental and subject to change, depending on its reception by the community. We will introduce a public and stable API if there is significant demand for it.\n\nPEFT supports an experimental API for custom module types for LoRA. Let's assume you have a LoRA implementation for LSTMs. Normally, you would not be able to tell PEFT to use it, even if it would theoretically work with PEFT. However, this is possible with dynamic dispatch of custom layers.\n\nThe experimental API currently looks like this:\n\n```python\nclass MyLoraLSTMLayer:\n    ...\n\nbase_model = ...  # load the base model that uses LSTMs\n\n# add the LSTM layer names to target_modules\nconfig = LoraConfig(..., target_modules=[\"lstm\"])\n# define a mapping from base layer type to LoRA layer type\ncustom_module_mapping = {nn.LSTM: MyLoraLSTMLayer}\n# register the new mapping\nconfig._register_custom_module(custom_module_mapping)\n# after registration, create the PEFT model\npeft_model = get_peft_model(base_model, config)\n# do training\n```\n\n<Tip>\n\nWhen you call [`get_peft_model`], you will see a warning because PEFT does not recognize the targeted module type. In this case, you can ignore this warning.\n\n</Tip>\n\nBy supplying a custom mapping, PEFT first checks the base model's layers against the custom mapping and dispatches to the custom LoRA layer type if there is a match. If there is no match, PEFT checks the built-in LoRA layer types for a match.\n\nTherefore, this feature can also be used to override existing dispatch logic, e.g. if you want to use your own LoRA layer for `nn.Linear` instead of using the one provided by PEFT.\n\nWhen creating your custom LoRA module, please follow the same rules as the [existing LoRA modules](https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora/layer.py). Some important constraints to consider:\n\n- The custom module should inherit from `nn.Module` and `peft.tuners.lora.layer.LoraLayer`.\n- The `__init__` method of the custom module should have the positional arguments `base_layer` and `adapter_name`. After this, there are additional `**kwargs` that you are free to use or ignore.\n- The learnable parameters should be stored in an `nn.ModuleDict` or `nn.ParameterDict`, where the key corresponds to the name of the specific adapter (remember that a model can have more than one adapter at a time).\n- The name of these learnable parameter attributes should start with `\"lora_\"`, e.g. `self.lora_new_param = ...`.\n- Some methods are optional, e.g. you only need to implement `merge` and `unmerge` if you want to support weight merging.\n\nCurrently, the information about the custom module does not persist when you save the model. When loading the model, you have to register the custom modules again."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: 09c06682-3d86-4316-8d85-11cc07712a06
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/developer_guides/custom_models", "title": "Custom models", "tokens": 3842, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"09c06682-3d86-4316-8d85-11cc07712a06\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/custom_models\", \"title\": \"Custom models\", \"tokens\": 3842, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"587b2ffc-a94b-4f12-a621-18f994f432f3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/custom_models\", \"title\": \"Custom models\", \"tokens\": 3842, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"f07eb5625e8d53aa533339337f7a581384cdf5d991cc351a3ce6956fbd60c213\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e582550d-a5e8-46a9-a10d-827048c1b4c8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/custom_models\", \"title\": \"Custom models\", \"tokens\": 3842, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"38fae3f1845dcc84639185279b37bea16ec2d7953acc3350311dbcac2d1c1b01\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12483, \"end_char_idx\": 14907, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "587b2ffc-a94b-4f12-a621-18f994f432f3", "doc_id": "587b2ffc-a94b-4f12-a621-18f994f432f3", "ref_doc_id": "587b2ffc-a94b-4f12-a621-18f994f432f3", "chroma:document": "In this case, you can ignore this warning.\n\n</Tip>\n\nBy supplying a custom mapping, PEFT first checks the base model's layers against the custom mapping and dispatches to the custom LoRA layer type if there is a match. If there is no match, PEFT checks the built-in LoRA layer types for a match.\n\nTherefore, this feature can also be used to override existing dispatch logic, e.g. if you want to use your own LoRA layer for `nn.Linear` instead of using the one provided by PEFT.\n\nWhen creating your custom LoRA module, please follow the same rules as the [existing LoRA modules](https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora/layer.py). Some important constraints to consider:\n\n- The custom module should inherit from `nn.Module` and `peft.tuners.lora.layer.LoraLayer`.\n- The `__init__` method of the custom module should have the positional arguments `base_layer` and `adapter_name`. After this, there are additional `**kwargs` that you are free to use or ignore.\n- The learnable parameters should be stored in an `nn.ModuleDict` or `nn.ParameterDict`, where the key corresponds to the name of the specific adapter (remember that a model can have more than one adapter at a time).\n- The name of these learnable parameter attributes should start with `\"lora_\"`, e.g. `self.lora_new_param = ...`.\n- Some methods are optional, e.g. you only need to implement `merge` and `unmerge` if you want to support weight merging.\n\nCurrently, the information about the custom module does not persist when you save the model. When loading the model, you have to register the custom modules again.\n\n```python\n# saving works as always and includes the parameters of the custom modules\npeft_model.save_pretrained(<model-path>)\n\n# loading the model later:\nbase_model = ...\n# load the LoRA config that you saved earlier\nconfig = LoraConfig.from_pretrained(<model-path>)\n# register the custom module again, the same way as the first time\ncustom_module_mapping = {nn.LSTM: MyLoraLSTMLayer}\nconfig._register_custom_module(custom_module_mapping)\n# pass the config instance to from_pretrained:\npeft_model = PeftModel.from_pretrained(model, tmp_path / \"lora-custom-module\", config=config)\n```\n\nIf you use this feature and find it useful, or if you encounter problems, let us know by creating an issue or a discussion on GitHub. This allows us to estimate the demand for this feature and add a public API if it is sufficiently high."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: 401090fd-07cf-4d7c-a3ff-5de4bee47dbb
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/developer_guides/torch_compile", "title": "torch.compile", "tokens": 828, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"401090fd-07cf-4d7c-a3ff-5de4bee47dbb\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/torch_compile\", \"title\": \"torch.compile\", \"tokens\": 828, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b1673381-5700-412f-a894-08033da37be0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/torch_compile\", \"title\": \"torch.compile\", \"tokens\": 828, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"e9207286ad2262c8f980020a5d659ab9ca39ca90a4a8b31ce23b9230ec8652cd\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2792, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b1673381-5700-412f-a894-08033da37be0", "doc_id": "b1673381-5700-412f-a894-08033da37be0", "ref_doc_id": "b1673381-5700-412f-a894-08033da37be0", "chroma:document": "# torch.compile\n\nIn PEFT, [torch.compile](https://pytorch.org/tutorials/intermediate/torch_compile_tutorial.html) works for some but not all features. The reason why it won't always work is because PEFT is highly dynamic in certain places (loading and switching between multiple adapters, for instance), which can cause trouble for `torch.compile`. In other places, `torch.compile` may work, but won't be as fast as expected because of graph breaks.\n\nIf you don't see an error, it doesn't necessarily mean that `torch.compile` worked correctly. It might give you an output, but the output is incorrect. This guide describes what works with `torch.compile` and what doesn't.\n\n> [!TIP]\n> Unless indicated otherwise, the default `torch.compile` settings were used.\n\n## Training and inference with `torch.compile`\n\nThese features **work** with `torch.compile`. Everything listed below was tested with a causal LM:\n\n- Training with `Trainer` from \ud83e\udd17 transformers\n- Training with a custom PyTorch loop\n- Inference\n- Generation\n\nThe following adapters were tested successfully:\n\n- AdaLoRA\n- BOFT\n- IA\u00b3\n- Layer Norm Tuning\n- LoHa\n- LoRA\n- LoRA + DoRA\n- OFT\n- VeRA\n- HRA\n\nThe following adapters **don't work** correctly for training or inference when using `torch.compile`:\n\n- LoKr\n- LoRA targeting embedding layers\n\n## Advanced PEFT features with `torch.compile`\n\nBelow are some of the more advanced PEFT features that **work**. They were all tested with LoRA.\n\n- `modules_to_save` (i.e. `config = LoraConfig(..., modules_to_save=...)`)\n- Merging adapters (one or multiple)\n- Merging multiple adapters into one adapter (i.e. calling `model.add_weighted_adapter(...)`)\n\nGenerally, we can expect that if a feature works correctly with LoRA and is also supported by other adapter types, it should also work for that adapter type.\n\nThe more advanced PEFT features below **don't work** in conjunction with `torch.compile`. Tests were run with LoRA:\n\n- Using PEFT adapters with quantization (bitsandbytes)\n- Inference with multiple adapters\n- Unloading (i.e. calling `model.merge_and_unload()`)\n- Disabling adapters (i.e. using `with model.disable_adapter()`)\n- Mixed adapter batches (i.e. calling `model(batch, adapter_names=[\"__base__\", \"default\", \"other\", ...])`)\n\n## Test cases\n\nAll the use cases listed above are tested inside of [`peft/tests/test_torch_compile.py`](https://github.com/huggingface/peft/blob/main/tests/test_torch_compile.py). If you want to check in more detail how we tested a certain feature, please go to that file and check the test that corresponds to your use case.\n\n> [!TIP]\n> If you have another use case where you know that `torch.compile` does or does not work with PEFT, please contribute by letting us know or by opening a PR to add this use case to the covered test cases."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: 7724efd9-5119-4490-985f-a716e744264d
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/developer_guides/checkpoint", "title": "PEFT checkpoint format", "tokens": 3329, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"7724efd9-5119-4490-985f-a716e744264d\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/checkpoint\", \"title\": \"PEFT checkpoint format\", \"tokens\": 3329, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"99d603d2-9073-4445-9161-4aa3803cd021\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/checkpoint\", \"title\": \"PEFT checkpoint format\", \"tokens\": 3329, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"b5a16eafb9f7797e65ffc2b56e056a1313df363eff570e9b50f1b7654f395a7f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5658df95-1e15-4d7c-995d-b4d4abb7096b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"46c15d137e660365839ab7b14e8558f2b68871f3703ab6ab171aa0e654f6098e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3252, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "99d603d2-9073-4445-9161-4aa3803cd021", "doc_id": "99d603d2-9073-4445-9161-4aa3803cd021", "ref_doc_id": "99d603d2-9073-4445-9161-4aa3803cd021", "chroma:document": "# PEFT checkpoint format\n\nThis document describes how PEFT's checkpoint files are structured and how to convert between the PEFT format and other formats.\n\n## PEFT files\n\nPEFT (parameter-efficient fine-tuning) methods only update a small subset of a model's parameters rather than all of them. This is nice because checkpoint files can generally be much smaller than the original model files and are easier to store and share. However, this also means that to load a PEFT model, you need to have the original model available as well.\n\nWhen you call [`~PeftModel.save_pretrained`] on a PEFT model, the PEFT model saves three files, described below:\n\n1. `adapter_model.safetensors` or `adapter_model.bin`\n\nBy default, the model is saved in the `safetensors` format, a secure alternative to the `bin` format, which is known to be susceptible to [security vulnerabilities](https://huggingface.co/docs/hub/security-pickle) because it uses the pickle utility under the hood. Both formats store the same `state_dict` though, and are interchangeable.\n\nThe `state_dict` only contains the parameters of the adapter module, not the base model. To illustrate the difference in size, a normal BERT model requires ~420MB of disk space, whereas an IA\u00b3 adapter on top of this BERT model only requires ~260KB.\n\n2. `adapter_config.json`\n\nThe `adapter_config.json` file contains the configuration of the adapter module, which is necessary to load the model. Below is an example of an `adapter_config.json` for an IA\u00b3 adapter with standard settings applied to a BERT model:\n\n```json\n{\n  \"auto_mapping\": {\n    \"base_model_class\": \"BertModel\",\n    \"parent_library\": \"transformers.models.bert.modeling_bert\"\n  },\n  \"base_model_name_or_path\": \"bert-base-uncased\",\n  \"fan_in_fan_out\": false,\n  \"feedforward_modules\": [\n    \"output.dense\"\n  ],\n  \"inference_mode\": true,\n  \"init_ia3_weights\": true,\n  \"modules_to_save\": null,\n  \"peft_type\": \"IA3\",\n  \"revision\": null,\n  \"target_modules\": [\n    \"key\",\n    \"value\",\n    \"output.dense\"\n  ],\n  \"task_type\": null\n}\n```\n\nThe configuration file contains:\n\n- the adapter module type stored, `\"peft_type\": \"IA3\"`\n- information about the base model like `\"base_model_name_or_path\": \"bert-base-uncased\"`\n- the revision of the model (if any), `\"revision\": null`\n\nIf the base model is not a pretrained Transformers model, the latter two entries will be `null`. Other than that, the settings are all related to the specific IA\u00b3 adapter that was used to fine-tune the model.\n\n3. `README.md`\n\nThe generated `README.md` is the model card of a PEFT model and contains a few pre-filled entries. The intent of this is to make it easier to share the model with others and to provide some basic information about the model. This file is not needed to load the model.\n\n## Convert to PEFT format\n\nWhen converting from another format to the PEFT format, we require both the `adapter_model.safetensors` (or `adapter_model.bin`) file and the `adapter_config.json` file.\n\n### adapter_model\n\nFor the model weights, it is important to use the correct mapping from parameter name to value for PEFT to load the file. Getting this mapping right is an exercise in checking the implementation details, as there is no generally agreed upon format for PEFT adapters."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: 5658df95-1e15-4d7c-995d-b4d4abb7096b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/developer_guides/checkpoint", "title": "PEFT checkpoint format", "tokens": 3329, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"5658df95-1e15-4d7c-995d-b4d4abb7096b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/checkpoint\", \"title\": \"PEFT checkpoint format\", \"tokens\": 3329, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"99d603d2-9073-4445-9161-4aa3803cd021\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/checkpoint\", \"title\": \"PEFT checkpoint format\", \"tokens\": 3329, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"b5a16eafb9f7797e65ffc2b56e056a1313df363eff570e9b50f1b7654f395a7f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7724efd9-5119-4490-985f-a716e744264d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/checkpoint\", \"title\": \"PEFT checkpoint format\", \"tokens\": 3329, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"036aa092c3a8f0019354d2360147339e9b86dfd2f2aa9c5f71f051a67a79c22d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"30fa40f0-a58a-4c0c-98cb-763e8f994898\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b3331765c8d40943785f02cac1b5642168a59d901d3e4baf2ba7665ee4878133\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2371, \"end_char_idx\": 5238, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "99d603d2-9073-4445-9161-4aa3803cd021", "doc_id": "99d603d2-9073-4445-9161-4aa3803cd021", "ref_doc_id": "99d603d2-9073-4445-9161-4aa3803cd021", "chroma:document": "Other than that, the settings are all related to the specific IA\u00b3 adapter that was used to fine-tune the model.\n\n3. `README.md`\n\nThe generated `README.md` is the model card of a PEFT model and contains a few pre-filled entries. The intent of this is to make it easier to share the model with others and to provide some basic information about the model. This file is not needed to load the model.\n\n## Convert to PEFT format\n\nWhen converting from another format to the PEFT format, we require both the `adapter_model.safetensors` (or `adapter_model.bin`) file and the `adapter_config.json` file.\n\n### adapter_model\n\nFor the model weights, it is important to use the correct mapping from parameter name to value for PEFT to load the file. Getting this mapping right is an exercise in checking the implementation details, as there is no generally agreed upon format for PEFT adapters.\n\nFortunately, figuring out this mapping is not overly complicated for common base cases. Let's look at a concrete example, the [`LoraLayer`](https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora/layer.py):\n\n```python\n# showing only part of the code\n\nclass LoraLayer(BaseTunerLayer):\n    # All names of layers that may contain (trainable) adapter weights\n    adapter_layer_names = (\"lora_A\", \"lora_B\", \"lora_embedding_A\", \"lora_embedding_B\")\n    # All names of other parameters that may contain adapter-related parameters\n    other_param_names = (\"r\", \"lora_alpha\", \"scaling\", \"lora_dropout\")\n\n    def __init__(self, base_layer: nn.Module, **kwargs) -> None:\n        self.base_layer = base_layer\n        self.r = {}\n        self.lora_alpha = {}\n        self.scaling = {}\n        self.lora_dropout = nn.ModuleDict({})\n        self.lora_A = nn.ModuleDict({})\n        self.lora_B = nn.ModuleDict({})\n        # For Embedding layer\n        self.lora_embedding_A = nn.ParameterDict({})\n        self.lora_embedding_B = nn.ParameterDict({})\n        # Mark the weight as unmerged\n        self._disable_adapters = False\n        self.merged_adapters = []\n        self.use_dora: dict[str, bool] = {}\n        self.lora_magnitude_vector: Optional[torch.nn.ParameterDict] = None  # for DoRA\n        self._caches: dict[str, Any] = {}\n        self.kwargs = kwargs\n```\n\nIn the `__init__` code used by all `LoraLayer` classes in PEFT, there are a bunch of parameters used to initialize the model, but only a few are relevant for the checkpoint file: `lora_A`, `lora_B`, `lora_embedding_A`, and `lora_embedding_B`. These parameters are listed in the class attribute `adapter_layer_names` and contain the learnable parameters, so they must be included in the checkpoint file. All the other parameters, like the rank `r`, are derived from the `adapter_config.json` and must be included there (unless the default value is used).\n\nLet's check the `state_dict` of a PEFT LoRA model applied to BERT."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: 30fa40f0-a58a-4c0c-98cb-763e8f994898
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/developer_guides/checkpoint", "title": "PEFT checkpoint format", "tokens": 3329, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"30fa40f0-a58a-4c0c-98cb-763e8f994898\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/checkpoint\", \"title\": \"PEFT checkpoint format\", \"tokens\": 3329, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"99d603d2-9073-4445-9161-4aa3803cd021\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/checkpoint\", \"title\": \"PEFT checkpoint format\", \"tokens\": 3329, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"b5a16eafb9f7797e65ffc2b56e056a1313df363eff570e9b50f1b7654f395a7f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5658df95-1e15-4d7c-995d-b4d4abb7096b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/checkpoint\", \"title\": \"PEFT checkpoint format\", \"tokens\": 3329, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"7d92964105909aa48c68734f48fbde7f73c1b13d7c2f702bc95e5cde9f318826\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"79488c27-0189-4259-8b46-629da898a3ea\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b347cb92f736563ad20f7ec4f14fe4b6341be9f918d66d57f70cb6bf71e86313\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4860, \"end_char_idx\": 7984, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "99d603d2-9073-4445-9161-4aa3803cd021", "doc_id": "99d603d2-9073-4445-9161-4aa3803cd021", "ref_doc_id": "99d603d2-9073-4445-9161-4aa3803cd021", "chroma:document": "These parameters are listed in the class attribute `adapter_layer_names` and contain the learnable parameters, so they must be included in the checkpoint file. All the other parameters, like the rank `r`, are derived from the `adapter_config.json` and must be included there (unless the default value is used).\n\nLet's check the `state_dict` of a PEFT LoRA model applied to BERT. When printing the first five keys using the default LoRA settings (the remaining keys are the same, just with different layer numbers), we get:\n\n- `base_model.model.encoder.layer.0.attention.self.query.lora_A.weight` \n- `base_model.model.encoder.layer.0.attention.self.query.lora_B.weight` \n- `base_model.model.encoder.layer.0.attention.self.value.lora_A.weight` \n- `base_model.model.encoder.layer.0.attention.self.value.lora_B.weight` \n- `base_model.model.encoder.layer.1.attention.self.query.lora_A.weight`\n- etc.\n\nLet's break this down:\n\n- By default, for BERT models, LoRA is applied to the `query` and `value` layers of the attention module. This is why you see `attention.self.query` and `attention.self.value` in the key names for each layer.\n- LoRA decomposes the weights into two low-rank matrices, `lora_A` and `lora_B`. This is where `lora_A` and `lora_B` come from in the key names.\n- These LoRA matrices are implemented as `nn.Linear` layers, so the parameters are stored in the `.weight` attribute (`lora_A.weight`, `lora_B.weight`).\n- By default, LoRA isn't applied to BERT's embedding layer, so there are _no entries_ for `lora_A_embedding` and `lora_B_embedding`.\n- The keys of the `state_dict` always start with `\"base_model.model.\"`. The reason is that, in PEFT, we wrap the base model inside a tuner-specific model (`LoraModel` in this case), which itself is wrapped in a general PEFT model (`PeftModel`). For this reason, these two prefixes are added to the keys. When converting to the PEFT format, it is required to add these prefixes.\n\n<Tip>\n\nThis last point is not true for prefix tuning techniques like prompt tuning. There, the extra embeddings are directly stored in the `state_dict` without any prefixes added to the keys.\n\n</Tip>\n\nWhen inspecting the parameter names in the loaded model, you might be surprised to find that they look a bit different, e.g. `base_model.model.encoder.layer.0.attention.self.query.lora_A.default.weight`. The difference is the *`.default`* part in the second to last segment. This part exists because PEFT generally allows the addition of multiple adapters at once (using an `nn.ModuleDict` or `nn.ParameterDict` to store them). For example, if you add another adapter called \"other\", the key for that adapter would be `base_model.model.encoder.layer.0.attention.self.query.lora_A.other.weight`.\n\nWhen you call [`~PeftModel.save_pretrained`], the adapter name is stripped from the keys. The reason is that the adapter name is not an important part of the model architecture; it is just an arbitrary name. When loading the adapter, you could choose a totally different name, and the model would still work the same way. This is why the adapter name is not stored in the checkpoint file."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: 79488c27-0189-4259-8b46-629da898a3ea
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/developer_guides/checkpoint", "title": "PEFT checkpoint format", "tokens": 3329, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"79488c27-0189-4259-8b46-629da898a3ea\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/checkpoint\", \"title\": \"PEFT checkpoint format\", \"tokens\": 3329, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"99d603d2-9073-4445-9161-4aa3803cd021\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/checkpoint\", \"title\": \"PEFT checkpoint format\", \"tokens\": 3329, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"b5a16eafb9f7797e65ffc2b56e056a1313df363eff570e9b50f1b7654f395a7f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"30fa40f0-a58a-4c0c-98cb-763e8f994898\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/checkpoint\", \"title\": \"PEFT checkpoint format\", \"tokens\": 3329, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"62d65183e3728172fe10018f0b5c59ea1eaf254345d33bf72782f7d9efcc6bea\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7524c0a4-5c39-4a15-b0d4-f86a0623d78e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5acd588aaa9ae4bd42b665c39338b4e4dfd860f27f6ad29573828d865799e6cd\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6287, \"end_char_idx\": 8890, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "99d603d2-9073-4445-9161-4aa3803cd021", "doc_id": "99d603d2-9073-4445-9161-4aa3803cd021", "ref_doc_id": "99d603d2-9073-4445-9161-4aa3803cd021", "chroma:document": "- By default, LoRA isn't applied to BERT's embedding layer, so there are _no entries_ for `lora_A_embedding` and `lora_B_embedding`.\n- The keys of the `state_dict` always start with `\"base_model.model.\"`. The reason is that, in PEFT, we wrap the base model inside a tuner-specific model (`LoraModel` in this case), which itself is wrapped in a general PEFT model (`PeftModel`). For this reason, these two prefixes are added to the keys. When converting to the PEFT format, it is required to add these prefixes.\n\n<Tip>\n\nThis last point is not true for prefix tuning techniques like prompt tuning. There, the extra embeddings are directly stored in the `state_dict` without any prefixes added to the keys.\n\n</Tip>\n\nWhen inspecting the parameter names in the loaded model, you might be surprised to find that they look a bit different, e.g. `base_model.model.encoder.layer.0.attention.self.query.lora_A.default.weight`. The difference is the *`.default`* part in the second to last segment. This part exists because PEFT generally allows the addition of multiple adapters at once (using an `nn.ModuleDict` or `nn.ParameterDict` to store them). For example, if you add another adapter called \"other\", the key for that adapter would be `base_model.model.encoder.layer.0.attention.self.query.lora_A.other.weight`.\n\nWhen you call [`~PeftModel.save_pretrained`], the adapter name is stripped from the keys. The reason is that the adapter name is not an important part of the model architecture; it is just an arbitrary name. When loading the adapter, you could choose a totally different name, and the model would still work the same way. This is why the adapter name is not stored in the checkpoint file.\n\n<Tip>\n\nIf you call `save_pretrained(\"some/path\")` and the adapter name is not `\"default\"`, the adapter is stored in a sub-directory with the same name as the adapter. So if the name is \"other\", it would be stored inside of `some/path/other`.\n\n</Tip>\n\nIn some circumstances, deciding which values to add to the checkpoint file can become a bit more complicated. For example, in PEFT, DoRA is implemented as a special case of LoRA. If you want to convert a DoRA model to PEFT, you should create a LoRA checkpoint with extra entries for DoRA. You can see this in the `__init__` of the previous `LoraLayer` code:\n\n```python\nself.lora_magnitude_vector: Optional[torch.nn.ParameterDict] = None  # for DoRA\n```\n\nThis indicates that there is an optional extra parameter per layer for DoRA.\n\n### adapter_config\n\nAll the other information needed to load a PEFT model is contained in the `adapter_config.json` file."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: 7524c0a4-5c39-4a15-b0d4-f86a0623d78e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/developer_guides/checkpoint", "title": "PEFT checkpoint format", "tokens": 3329, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"7524c0a4-5c39-4a15-b0d4-f86a0623d78e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/checkpoint\", \"title\": \"PEFT checkpoint format\", \"tokens\": 3329, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"99d603d2-9073-4445-9161-4aa3803cd021\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/checkpoint\", \"title\": \"PEFT checkpoint format\", \"tokens\": 3329, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"b5a16eafb9f7797e65ffc2b56e056a1313df363eff570e9b50f1b7654f395a7f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"79488c27-0189-4259-8b46-629da898a3ea\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/checkpoint\", \"title\": \"PEFT checkpoint format\", \"tokens\": 3329, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"71b9f5f8e1a1ce36c75f41bc948cf7dbc774ccc52defdd2a23eaa07e43fc3276\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6f8c60ab-8913-4341-a69e-4c44b6a87f74\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f4ebd25dc46ccd6211ee4a3c6dc94c393b087fbb262904920babf7a0e0c676f2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7204, \"end_char_idx\": 10217, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "99d603d2-9073-4445-9161-4aa3803cd021", "doc_id": "99d603d2-9073-4445-9161-4aa3803cd021", "ref_doc_id": "99d603d2-9073-4445-9161-4aa3803cd021", "chroma:document": "The difference is the *`.default`* part in the second to last segment. This part exists because PEFT generally allows the addition of multiple adapters at once (using an `nn.ModuleDict` or `nn.ParameterDict` to store them). For example, if you add another adapter called \"other\", the key for that adapter would be `base_model.model.encoder.layer.0.attention.self.query.lora_A.other.weight`.\n\nWhen you call [`~PeftModel.save_pretrained`], the adapter name is stripped from the keys. The reason is that the adapter name is not an important part of the model architecture; it is just an arbitrary name. When loading the adapter, you could choose a totally different name, and the model would still work the same way. This is why the adapter name is not stored in the checkpoint file.\n\n<Tip>\n\nIf you call `save_pretrained(\"some/path\")` and the adapter name is not `\"default\"`, the adapter is stored in a sub-directory with the same name as the adapter. So if the name is \"other\", it would be stored inside of `some/path/other`.\n\n</Tip>\n\nIn some circumstances, deciding which values to add to the checkpoint file can become a bit more complicated. For example, in PEFT, DoRA is implemented as a special case of LoRA. If you want to convert a DoRA model to PEFT, you should create a LoRA checkpoint with extra entries for DoRA. You can see this in the `__init__` of the previous `LoraLayer` code:\n\n```python\nself.lora_magnitude_vector: Optional[torch.nn.ParameterDict] = None  # for DoRA\n```\n\nThis indicates that there is an optional extra parameter per layer for DoRA.\n\n### adapter_config\n\nAll the other information needed to load a PEFT model is contained in the `adapter_config.json` file. Let's check this file for a LoRA model applied to BERT:\n\n```json\n{\n  \"alpha_pattern\": {},\n  \"auto_mapping\": {\n    \"base_model_class\": \"BertModel\",\n    \"parent_library\": \"transformers.models.bert.modeling_bert\"\n  },\n  \"base_model_name_or_path\": \"bert-base-uncased\",\n  \"bias\": \"none\",\n  \"fan_in_fan_out\": false,\n  \"inference_mode\": true,\n  \"init_lora_weights\": true,\n  \"layer_replication\": null,\n  \"layers_pattern\": null,\n  \"layers_to_transform\": null,\n  \"loftq_config\": {},\n  \"lora_alpha\": 8,\n  \"lora_dropout\": 0.0,\n  \"megatron_config\": null,\n  \"megatron_core\": \"megatron.core\",\n  \"modules_to_save\": null,\n  \"peft_type\": \"LORA\",\n  \"r\": 8,\n  \"rank_pattern\": {},\n  \"revision\": null,\n  \"target_modules\": [\n    \"query\",\n    \"value\"\n  ],\n  \"task_type\": null,\n  \"use_dora\": false,\n  \"use_rslora\": false\n}\n```\n\nThis contains a lot of entries, and at first glance, it could feel overwhelming to figure out all the right values to put in there. However, most of the entries are not necessary to load the model. This is either because they use the default values and don't need to be added or because they only affect the initialization of the LoRA weights, which is irrelevant when it comes to loading the model. If you find that you don't know what a specific parameter does, e.g., `\"use_rslora\",` don't add it, and you should be fine."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: 6f8c60ab-8913-4341-a69e-4c44b6a87f74
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/developer_guides/checkpoint", "title": "PEFT checkpoint format", "tokens": 3329, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"6f8c60ab-8913-4341-a69e-4c44b6a87f74\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/checkpoint\", \"title\": \"PEFT checkpoint format\", \"tokens\": 3329, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"99d603d2-9073-4445-9161-4aa3803cd021\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/checkpoint\", \"title\": \"PEFT checkpoint format\", \"tokens\": 3329, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"b5a16eafb9f7797e65ffc2b56e056a1313df363eff570e9b50f1b7654f395a7f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7524c0a4-5c39-4a15-b0d4-f86a0623d78e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/checkpoint\", \"title\": \"PEFT checkpoint format\", \"tokens\": 3329, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"bca7f26c0ea2aebd9ef7a5a8478610b858ed6f15b13c0d56b151beffb7d7996c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"04af9a16-a8a7-4004-923c-1d392e26bb9e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e064b178a5b5a4148beb15ff363677516edd059f7719afc0e85281b819b2717c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8891, \"end_char_idx\": 11981, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "99d603d2-9073-4445-9161-4aa3803cd021", "doc_id": "99d603d2-9073-4445-9161-4aa3803cd021", "ref_doc_id": "99d603d2-9073-4445-9161-4aa3803cd021", "chroma:document": "Let's check this file for a LoRA model applied to BERT:\n\n```json\n{\n  \"alpha_pattern\": {},\n  \"auto_mapping\": {\n    \"base_model_class\": \"BertModel\",\n    \"parent_library\": \"transformers.models.bert.modeling_bert\"\n  },\n  \"base_model_name_or_path\": \"bert-base-uncased\",\n  \"bias\": \"none\",\n  \"fan_in_fan_out\": false,\n  \"inference_mode\": true,\n  \"init_lora_weights\": true,\n  \"layer_replication\": null,\n  \"layers_pattern\": null,\n  \"layers_to_transform\": null,\n  \"loftq_config\": {},\n  \"lora_alpha\": 8,\n  \"lora_dropout\": 0.0,\n  \"megatron_config\": null,\n  \"megatron_core\": \"megatron.core\",\n  \"modules_to_save\": null,\n  \"peft_type\": \"LORA\",\n  \"r\": 8,\n  \"rank_pattern\": {},\n  \"revision\": null,\n  \"target_modules\": [\n    \"query\",\n    \"value\"\n  ],\n  \"task_type\": null,\n  \"use_dora\": false,\n  \"use_rslora\": false\n}\n```\n\nThis contains a lot of entries, and at first glance, it could feel overwhelming to figure out all the right values to put in there. However, most of the entries are not necessary to load the model. This is either because they use the default values and don't need to be added or because they only affect the initialization of the LoRA weights, which is irrelevant when it comes to loading the model. If you find that you don't know what a specific parameter does, e.g., `\"use_rslora\",` don't add it, and you should be fine. Also note that as more options are added, this file will get more entries in the future, but it should be backward compatible.\n\nAt the minimum, you should include the following entries:\n\n```json\n{\n  \"target_modules\": [\"query\", \"value\"],\n  \"peft_type\": \"LORA\"\n}\n```\n\nHowever, adding as many entries as possible, like the rank `r` or the `base_model_name_or_path` (if it's a Transformers model) is recommended. This information can help others understand the model better and share it more easily. To check which keys and values are expected, check out the [config.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora/config.py) file (as an example, this is the config file for LoRA) in the PEFT source code.\n\n## Model storage\n\nIn some circumstances, you might want to store the whole PEFT model, including the base weights. This can be necessary if, for instance, the base model is not available to the users trying to load the PEFT model. You can merge the weights first or convert it into a Transformer model.\n\n### Merge the weights\n\nThe most straightforward way to store the whole PEFT model is to merge the adapter weights into the base weights:\n\n```python\nmerged_model = model.merge_and_unload()\nmerged_model.save_pretrained(...)\n```\n\nThere are some disadvantages to this approach, though:\n\n- Once [`~LoraModel.merge_and_unload`] is called, you get a basic model without any PEFT-specific functionality. This means you can't use any of the PEFT-specific methods anymore.\n- You cannot unmerge the weights, load multiple adapters at once, disable the adapter, etc.\n- Not all PEFT methods support merging weights.\n- Some PEFT methods may generally allow merging, but not with specific settings (e.g. when using certain quantization techniques)."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: 04af9a16-a8a7-4004-923c-1d392e26bb9e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/developer_guides/checkpoint", "title": "PEFT checkpoint format", "tokens": 3329, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"04af9a16-a8a7-4004-923c-1d392e26bb9e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/checkpoint\", \"title\": \"PEFT checkpoint format\", \"tokens\": 3329, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"99d603d2-9073-4445-9161-4aa3803cd021\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/checkpoint\", \"title\": \"PEFT checkpoint format\", \"tokens\": 3329, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"b5a16eafb9f7797e65ffc2b56e056a1313df363eff570e9b50f1b7654f395a7f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6f8c60ab-8913-4341-a69e-4c44b6a87f74\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/checkpoint\", \"title\": \"PEFT checkpoint format\", \"tokens\": 3329, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"347709a5226a6fd79b522ae68fea5b6b96e7cdd729a05e0f46e20a38b7778587\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10346, \"end_char_idx\": 13198, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "99d603d2-9073-4445-9161-4aa3803cd021", "doc_id": "99d603d2-9073-4445-9161-4aa3803cd021", "ref_doc_id": "99d603d2-9073-4445-9161-4aa3803cd021", "chroma:document": "At the minimum, you should include the following entries:\n\n```json\n{\n  \"target_modules\": [\"query\", \"value\"],\n  \"peft_type\": \"LORA\"\n}\n```\n\nHowever, adding as many entries as possible, like the rank `r` or the `base_model_name_or_path` (if it's a Transformers model) is recommended. This information can help others understand the model better and share it more easily. To check which keys and values are expected, check out the [config.py](https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora/config.py) file (as an example, this is the config file for LoRA) in the PEFT source code.\n\n## Model storage\n\nIn some circumstances, you might want to store the whole PEFT model, including the base weights. This can be necessary if, for instance, the base model is not available to the users trying to load the PEFT model. You can merge the weights first or convert it into a Transformer model.\n\n### Merge the weights\n\nThe most straightforward way to store the whole PEFT model is to merge the adapter weights into the base weights:\n\n```python\nmerged_model = model.merge_and_unload()\nmerged_model.save_pretrained(...)\n```\n\nThere are some disadvantages to this approach, though:\n\n- Once [`~LoraModel.merge_and_unload`] is called, you get a basic model without any PEFT-specific functionality. This means you can't use any of the PEFT-specific methods anymore.\n- You cannot unmerge the weights, load multiple adapters at once, disable the adapter, etc.\n- Not all PEFT methods support merging weights.\n- Some PEFT methods may generally allow merging, but not with specific settings (e.g. when using certain quantization techniques).\n- The whole model will be much larger than the PEFT model, as it will contain all the base weights as well.\n\nBut inference with a merged model should be a bit faster.\n\n### Convert to a Transformers model\n\nAnother way to save the whole model, assuming the base model is a Transformers model, is to use this hacky approach to directly insert the PEFT weights into the base model and save it, which only works if you \"trick\" Transformers into believing the PEFT model is not a PEFT model. This only works with LoRA because other adapters are not implemented in Transformers.\n\n```python\nmodel = ...  # the PEFT model\n...\n# after you finish training the model, save it in a temporary location\nmodel.save_pretrained(<temp_location>)\n# now load this model directly into a transformers model, without the PEFT wrapper\n# the PEFT weights are directly injected into the base model\nmodel_loaded = AutoModel.from_pretrained(<temp_location>)\n# now make the loaded model believe that it is _not_ a PEFT model\nmodel_loaded._hf_peft_config_loaded = False\n# now when we save it, it will save the whole model\nmodel_loaded.save_pretrained(<final_location>)\n# or upload to Hugging Face Hub\nmodel_loaded.push_to_hub(<final_location>)\n```"}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: 4d33b76d-4218-4e3a-b7e4-021ff3b44155
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/developer_guides/contributing", "title": "Contribute to PEFT", "tokens": 1440, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"4d33b76d-4218-4e3a-b7e4-021ff3b44155\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/contributing\", \"title\": \"Contribute to PEFT\", \"tokens\": 1440, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1062d1ad-11e2-4be1-9b6f-84d486f8b21d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/contributing\", \"title\": \"Contribute to PEFT\", \"tokens\": 1440, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"e2e224fcf3925fdbf7dd2f42ab395bae6749d9b56a9ab6b0d7360707a7fd39b0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"430a4225-1239-4617-9204-dd7fc6c15026\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d26228fe18e8883457ac0706139fd2caa9a4d96c8204dcffd07d4638e60152cc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3341, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1062d1ad-11e2-4be1-9b6f-84d486f8b21d", "doc_id": "1062d1ad-11e2-4be1-9b6f-84d486f8b21d", "ref_doc_id": "1062d1ad-11e2-4be1-9b6f-84d486f8b21d", "chroma:document": "# Contribute to PEFT\n\nWe are happy to accept contributions to PEFT. If you plan to contribute, please read this to make the process as smooth as possible.\n\n## Installation\n\nFor code contributions to PEFT, you should choose the [\"source\"](../install#source) installation method.\n\nIf you are new to creating a pull request, follow the [Creating a pull request](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request) guide by GitHub.\n\n## Tests and code quality checks\n\nRegardless of the contribution type (unless it\u2019s only about the docs), you should run tests and code quality checks before creating a PR to ensure your contribution doesn\u2019t break anything and follows the project standards.\n\nWe provide a Makefile to execute the necessary tests. Run the code below for the unit test:\n\n```sh\nmake test\n```\n\nRun one of the following to either only check or check and fix code quality and style:\n\n```sh\nmake quality  # just check\nmake style  # check and fix\n```\n\nYou can also set up [`pre-commit`](https://pre-commit.com/) to run these fixes\nautomatically as Git commit hooks.\n\n```bash\n$ pip install pre-commit\n$ pre-commit install\n```\n\nRunning all the tests can take a couple of minutes, so during development it can be more efficient to only run tests specific to your change:\n\n```sh\npytest tests/ -k <name-of-test>\n```\n\nThis should finish much quicker and allow for faster iteration. However, you should still run the whole test suite before creating a PR because your change can inadvertently break tests that at first glance are unrelated.\n\nIf your change is specific to a hardware setting (e.g., it requires CUDA), take a look at [tests/test_gpu_examples.py](https://github.com/huggingface/peft/blob/1c1c7fdaa6e6abaa53939b865dee1eded82ad032/tests/test_gpu_examples.py) and [tests/test_common_gpu.py](https://github.com/huggingface/peft/blob/1c1c7fdaa6e6abaa53939b865dee1eded82ad032/tests/test_common_gpu.py) to see if it makes sense to add tests there. If your change could have an effect on saving and loading models, please run the tests with the `--regression` flag to trigger regression tests.\n\nIt can happen that while you\u2019re working on your PR, the underlying code base changes due to other changes being merged. If that happens \u2013 especially when there is a merge conflict \u2013 please update your branch with the latest changes. This can be a merge or a rebase, and we'll squash and merge the PR once it\u2019s ready.\n\n## PR description\n\nWhen opening a PR, please provide a nice description of the change you're proposing. If it relates to other issues or PRs, please reference them. Providing a good description not only helps the reviewers review your code better and faster, it can also be used later (as a basis) for the commit message which helps with long term maintenance of the project.\n\nIf your code makes some non-trivial changes, it may also be a good idea to add comments to the code to explain those changes. For example, if you had to iterate on your implementation multiple times because the most obvious way didn\u2019t work, it\u2019s a good indication that a code comment is needed.\n\n## Bugfixes\n\nPlease give a description of the circumstances that led to the bug. If there is an existing issue, please link to it (e.g., \u201cResolves #12345\u201d)."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: 430a4225-1239-4617-9204-dd7fc6c15026
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/developer_guides/contributing", "title": "Contribute to PEFT", "tokens": 1440, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"430a4225-1239-4617-9204-dd7fc6c15026\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/contributing\", \"title\": \"Contribute to PEFT\", \"tokens\": 1440, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1062d1ad-11e2-4be1-9b6f-84d486f8b21d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/contributing\", \"title\": \"Contribute to PEFT\", \"tokens\": 1440, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"e2e224fcf3925fdbf7dd2f42ab395bae6749d9b56a9ab6b0d7360707a7fd39b0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4d33b76d-4218-4e3a-b7e4-021ff3b44155\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/contributing\", \"title\": \"Contribute to PEFT\", \"tokens\": 1440, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"d9e6e1fdc04861704e0781991b997df2d1b25014f9aecd7d61f2a819dd1631aa\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ee6f4f1f-9f03-4a80-b8d1-a1dccefb1514\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"58ed11205cbb8bb5c59ca172a99a3bd18e6df12364e057751a8faa6edac6bcca\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2048, \"end_char_idx\": 5581, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1062d1ad-11e2-4be1-9b6f-84d486f8b21d", "doc_id": "1062d1ad-11e2-4be1-9b6f-84d486f8b21d", "ref_doc_id": "1062d1ad-11e2-4be1-9b6f-84d486f8b21d", "chroma:document": "If your change could have an effect on saving and loading models, please run the tests with the `--regression` flag to trigger regression tests.\n\nIt can happen that while you\u2019re working on your PR, the underlying code base changes due to other changes being merged. If that happens \u2013 especially when there is a merge conflict \u2013 please update your branch with the latest changes. This can be a merge or a rebase, and we'll squash and merge the PR once it\u2019s ready.\n\n## PR description\n\nWhen opening a PR, please provide a nice description of the change you're proposing. If it relates to other issues or PRs, please reference them. Providing a good description not only helps the reviewers review your code better and faster, it can also be used later (as a basis) for the commit message which helps with long term maintenance of the project.\n\nIf your code makes some non-trivial changes, it may also be a good idea to add comments to the code to explain those changes. For example, if you had to iterate on your implementation multiple times because the most obvious way didn\u2019t work, it\u2019s a good indication that a code comment is needed.\n\n## Bugfixes\n\nPlease give a description of the circumstances that led to the bug. If there is an existing issue, please link to it (e.g., \u201cResolves #12345\u201d).\n\nIdeally when a bugfix is provided, it should be accompanied by a test for the bug. The test should fail with the current code and pass with the bugfix. Add a comment to the test that references the issue or PR. Without a test, it is more difficult to prevent regressions in the future.\n\n## Add a new fine-tuning method\n\nNew parameter-efficient fine-tuning methods are developed all the time. If you would like to add a new and promising method to PEFT, please follow these steps.\n\n1. Before you start to implement the new method, please open a GitHub issue with your proposal. This way, the maintainers can give you some early feedback.\n2. Please add a link to the source (usually a paper) of the method. Some evidence should be provided there is general interest in using the method. We will not add new methods that are freshly published, but there is no evidence of demand for it.\n3. When implementing the method, it makes sense to look for existing implementations that already exist as a guide. Moreover, when you structure your code, please take inspiration from the other PEFT methods. For example, if your method is similar to LoRA, it makes sense to structure your code similarly or even reuse some functions or classes where it makes sense (some code duplication is okay, but don\u2019t overdo it).\n4. Ideally, in addition to the implementation of the new method, there should also be examples (notebooks, scripts), documentation, and an extensive test suite that proves the method works with a variety of tasks. However, this can be more challenging so it is acceptable to only provide the implementation and at least one working example. Documentation and tests can be added in follow up PRs.\n5. Once you have something that seems to be working, don\u2019t hesitate to create a draft PR even if it\u2019s not in a mergeable state yet. The maintainers are happy to give you feedback and guidance along the way.\n\n## Add other features\n\nIt is best if you first open an issue on GitHub with a proposal to add the new feature. This way, you can discuss with the maintainers if it makes sense to add the feature before spending too much time on implementing it.\n\nNew features should generally be accompanied by tests and documentation or examples."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: ee6f4f1f-9f03-4a80-b8d1-a1dccefb1514
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/developer_guides/contributing", "title": "Contribute to PEFT", "tokens": 1440, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"ee6f4f1f-9f03-4a80-b8d1-a1dccefb1514\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/contributing\", \"title\": \"Contribute to PEFT\", \"tokens\": 1440, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1062d1ad-11e2-4be1-9b6f-84d486f8b21d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/contributing\", \"title\": \"Contribute to PEFT\", \"tokens\": 1440, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"e2e224fcf3925fdbf7dd2f42ab395bae6749d9b56a9ab6b0d7360707a7fd39b0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"430a4225-1239-4617-9204-dd7fc6c15026\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/contributing\", \"title\": \"Contribute to PEFT\", \"tokens\": 1440, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"c8b1c5614a36ac9081491ea0909a6096ecc0f23209a88963d22fd6f007791704\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3735, \"end_char_idx\": 5830, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1062d1ad-11e2-4be1-9b6f-84d486f8b21d", "doc_id": "1062d1ad-11e2-4be1-9b6f-84d486f8b21d", "ref_doc_id": "1062d1ad-11e2-4be1-9b6f-84d486f8b21d", "chroma:document": "If you would like to add a new and promising method to PEFT, please follow these steps.\n\n1. Before you start to implement the new method, please open a GitHub issue with your proposal. This way, the maintainers can give you some early feedback.\n2. Please add a link to the source (usually a paper) of the method. Some evidence should be provided there is general interest in using the method. We will not add new methods that are freshly published, but there is no evidence of demand for it.\n3. When implementing the method, it makes sense to look for existing implementations that already exist as a guide. Moreover, when you structure your code, please take inspiration from the other PEFT methods. For example, if your method is similar to LoRA, it makes sense to structure your code similarly or even reuse some functions or classes where it makes sense (some code duplication is okay, but don\u2019t overdo it).\n4. Ideally, in addition to the implementation of the new method, there should also be examples (notebooks, scripts), documentation, and an extensive test suite that proves the method works with a variety of tasks. However, this can be more challenging so it is acceptable to only provide the implementation and at least one working example. Documentation and tests can be added in follow up PRs.\n5. Once you have something that seems to be working, don\u2019t hesitate to create a draft PR even if it\u2019s not in a mergeable state yet. The maintainers are happy to give you feedback and guidance along the way.\n\n## Add other features\n\nIt is best if you first open an issue on GitHub with a proposal to add the new feature. This way, you can discuss with the maintainers if it makes sense to add the feature before spending too much time on implementing it.\n\nNew features should generally be accompanied by tests and documentation or examples. Without the latter, users will have a hard time discovering your cool new feature.\n\nChanges to the code should be implemented in a backward-compatible way. For example, existing code should continue to work the same way after the feature is merged."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: 3863ee2b-dcd9-471b-9ee0-d2ce4184f738
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/developer_guides/low_level_api", "title": "Adapter injection", "tokens": 925, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"3863ee2b-dcd9-471b-9ee0-d2ce4184f738\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/low_level_api\", \"title\": \"Adapter injection\", \"tokens\": 925, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"278ddb10-0e84-4380-a9d6-337fd3d5b6e5\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/low_level_api\", \"title\": \"Adapter injection\", \"tokens\": 925, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"8c5c5ea776aaac2fa439ef2f6e3c6acebef075ea832f2eb32be99492e4462c4d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3022, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "278ddb10-0e84-4380-a9d6-337fd3d5b6e5", "doc_id": "278ddb10-0e84-4380-a9d6-337fd3d5b6e5", "ref_doc_id": "278ddb10-0e84-4380-a9d6-337fd3d5b6e5", "chroma:document": "# Adapter injection\n\nWith PEFT, you can inject trainable adapters into any `torch` module which allows you to use adapter methods without relying on the modeling classes in PEFT. Currently, PEFT supports injecting [LoRA](../conceptual_guides/adapter#low-rank-adaptation-lora), [AdaLoRA](../conceptual_guides/adapter#adaptive-low-rank-adaptation-adalora), and [IA3](../conceptual_guides/ia3) into models because for these adapters, inplace modification of the model is sufficient for finetuning it.\n\nCheck the table below to see when you should inject adapters.\n\n| Pros | Cons |\n|---|---|\n| the model is modified inplace, keeping all the original attributes and methods | manually write the `from_pretrained` and `save_pretrained` utility functions from Hugging Face to save and load adapters |\n| works for any `torch` module and modality | doesn't work with any of the utility methods provided by `PeftModel` such as disabling and merging adapters |\n\nTo perform the adapter injection, use the [`inject_adapter_in_model`] method. This method takes 3 arguments, the PEFT config, the model, and an optional adapter name. You can also attach multiple adapters to the model if you call [`inject_adapter_in_model`] multiple times with different adapter names.\n\nFor example, to inject LoRA adapters into the `linear` submodule of the `DummyModel` module:\n\n```python\nimport torch\nfrom peft import inject_adapter_in_model, LoraConfig\n\nclass DummyModel(torch.nn.Module):\n    def __init__(self):\n        super().__init__()\n        self.embedding = torch.nn.Embedding(10, 10)\n        self.linear = torch.nn.Linear(10, 10)\n        self.lm_head = torch.nn.Linear(10, 10)\n\n    def forward(self, input_ids):\n        x = self.embedding(input_ids)\n        x = self.linear(x)\n        x = self.lm_head(x)\n        return x\n\n\nlora_config = LoraConfig(\n    lora_alpha=16,\n    lora_dropout=0.1,\n    r=64,\n    bias=\"none\",\n    target_modules=[\"linear\"],\n)\n\nmodel = DummyModel()\nmodel = inject_adapter_in_model(lora_config, model)\n\ndummy_inputs = torch.LongTensor([[0, 1, 2, 3, 4, 5, 6, 7]])\ndummy_outputs = model(dummy_inputs)\n```\n\nPrint the model to see that the adapters have been correctly injected.\n\n```bash\nDummyModel(\n  (embedding): Embedding(10, 10)\n  (linear): Linear(\n    in_features=10, out_features=10, bias=True\n    (lora_dropout): ModuleDict(\n      (default): Dropout(p=0.1, inplace=False)\n    )\n    (lora_A): ModuleDict(\n      (default): Linear(in_features=10, out_features=64, bias=False)\n    )\n    (lora_B): ModuleDict(\n      (default): Linear(in_features=64, out_features=10, bias=False)\n    )\n    (lora_embedding_A): ParameterDict()\n    (lora_embedding_B): ParameterDict()\n  )\n  (lm_head): Linear(in_features=10, out_features=10, bias=True)\n)\n```\n\nTo only save the adapter, use the [`get_peft_model_state_dict`] function:\n\n```python\nfrom peft import get_peft_model_state_dict\n\npeft_state_dict = get_peft_model_state_dict(model)\nprint(peft_state_dict)\n```\n\nOtherwise, `model.state_dict()` returns the full state dict of the model."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: 670897a8-0b7e-4f71-b735-1f480d182273
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/developer_guides/quantization", "title": "Quantization", "tokens": 2385, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"670897a8-0b7e-4f71-b735-1f480d182273\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/quantization\", \"title\": \"Quantization\", \"tokens\": 2385, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"071f71de-9780-44e0-8fe6-0252113604f2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/quantization\", \"title\": \"Quantization\", \"tokens\": 2385, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"09bd4281c794e439948a282fe801dcbc6f8a70566ca7092c099c178e9bc863a3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"be9091d1-d91f-494d-a0f7-65b34c25a7cb\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1a9a7ebcb6ffe4028ccb09502d7f31923edbc186387036bc0a29c89d8cf66f2b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2924, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "071f71de-9780-44e0-8fe6-0252113604f2", "doc_id": "071f71de-9780-44e0-8fe6-0252113604f2", "ref_doc_id": "071f71de-9780-44e0-8fe6-0252113604f2", "chroma:document": "# Quantization\n\nQuantization represents data with fewer bits, making it a useful technique for reducing memory-usage and accelerating inference especially when it comes to large language models (LLMs). There are several ways to quantize a model including:\n\n* optimizing which model weights are quantized with the [AWQ](https://hf.co/papers/2306.00978) algorithm\n* independently quantizing each row of a weight matrix with the [GPTQ](https://hf.co/papers/2210.17323) algorithm\n* quantizing to 8-bit and 4-bit precision with the [bitsandbytes](https://github.com/TimDettmers/bitsandbytes) library\n* quantizing to as low as 2-bit precision with the [AQLM](https://arxiv.org/abs/2401.06118) algorithm\n\nHowever, after a model is quantized it isn't typically further trained for downstream tasks because training can be unstable due to the lower precision of the weights and activations. But since PEFT methods only add *extra* trainable parameters, this allows you to train a quantized model with a PEFT adapter on top! Combining quantization with PEFT can be a good strategy for training even the largest models on a single GPU. For example, [QLoRA](https://hf.co/papers/2305.14314) is a method that quantizes a model to 4-bits and then trains it with LoRA. This method allows you to finetune a 65B parameter model on a single 48GB GPU!\n\nIn this guide, you'll see how to quantize a model to 4-bits and train it with LoRA.\n\n## Quantize a model\n\n[bitsandbytes](https://github.com/TimDettmers/bitsandbytes) is a quantization library with a Transformers integration. With this integration, you can quantize a model to 8 or 4-bits and enable many other options by configuring the [`~transformers.BitsAndBytesConfig`] class. For example, you can:\n\n* set `load_in_4bit=True` to quantize the model to 4-bits when you load it\n* set `bnb_4bit_quant_type=\"nf4\"` to use a special 4-bit data type for weights initialized from a normal distribution\n* set `bnb_4bit_use_double_quant=True` to use a nested quantization scheme to quantize the already quantized weights\n* set `bnb_4bit_compute_dtype=torch.bfloat16` to use bfloat16 for faster computation\n\n```py\nimport torch\nfrom transformers import BitsAndBytesConfig\n\nconfig = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_quant_type=\"nf4\",\n    bnb_4bit_use_double_quant=True,\n    bnb_4bit_compute_dtype=torch.bfloat16,\n)\n```\n\nPass the `config` to the [`~transformers.AutoModelForCausalLM.from_pretrained`] method.\n\n```py\nfrom transformers import AutoModelForCausalLM\n\nmodel = AutoModelForCausalLM.from_pretrained(\"mistralai/Mistral-7B-v0.1\", quantization_config=config)\n```\n\nNext, you should call the [`~peft.utils.prepare_model_for_kbit_training`] function to preprocess the quantized model for training.\n\n```py\nfrom peft import prepare_model_for_kbit_training\n\nmodel = prepare_model_for_kbit_training(model)\n```\n\nNow that the quantized model is ready, let's set up a configuration."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: be9091d1-d91f-494d-a0f7-65b34c25a7cb
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/developer_guides/quantization", "title": "Quantization", "tokens": 2385, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"be9091d1-d91f-494d-a0f7-65b34c25a7cb\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/quantization\", \"title\": \"Quantization\", \"tokens\": 2385, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"071f71de-9780-44e0-8fe6-0252113604f2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/quantization\", \"title\": \"Quantization\", \"tokens\": 2385, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"09bd4281c794e439948a282fe801dcbc6f8a70566ca7092c099c178e9bc863a3\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"670897a8-0b7e-4f71-b735-1f480d182273\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/quantization\", \"title\": \"Quantization\", \"tokens\": 2385, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"46c8c357a2a28424251f315df3a6879843beba8b4e570708cc66d3cf488708dc\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4e5b791b-6862-462c-bdeb-a7eebc87375f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c226d5576c259d021250bd3ac3c5d186472d04749a0328145780f5f67627e376\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1419, \"end_char_idx\": 4373, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "071f71de-9780-44e0-8fe6-0252113604f2", "doc_id": "071f71de-9780-44e0-8fe6-0252113604f2", "ref_doc_id": "071f71de-9780-44e0-8fe6-0252113604f2", "chroma:document": "## Quantize a model\n\n[bitsandbytes](https://github.com/TimDettmers/bitsandbytes) is a quantization library with a Transformers integration. With this integration, you can quantize a model to 8 or 4-bits and enable many other options by configuring the [`~transformers.BitsAndBytesConfig`] class. For example, you can:\n\n* set `load_in_4bit=True` to quantize the model to 4-bits when you load it\n* set `bnb_4bit_quant_type=\"nf4\"` to use a special 4-bit data type for weights initialized from a normal distribution\n* set `bnb_4bit_use_double_quant=True` to use a nested quantization scheme to quantize the already quantized weights\n* set `bnb_4bit_compute_dtype=torch.bfloat16` to use bfloat16 for faster computation\n\n```py\nimport torch\nfrom transformers import BitsAndBytesConfig\n\nconfig = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_quant_type=\"nf4\",\n    bnb_4bit_use_double_quant=True,\n    bnb_4bit_compute_dtype=torch.bfloat16,\n)\n```\n\nPass the `config` to the [`~transformers.AutoModelForCausalLM.from_pretrained`] method.\n\n```py\nfrom transformers import AutoModelForCausalLM\n\nmodel = AutoModelForCausalLM.from_pretrained(\"mistralai/Mistral-7B-v0.1\", quantization_config=config)\n```\n\nNext, you should call the [`~peft.utils.prepare_model_for_kbit_training`] function to preprocess the quantized model for training.\n\n```py\nfrom peft import prepare_model_for_kbit_training\n\nmodel = prepare_model_for_kbit_training(model)\n```\n\nNow that the quantized model is ready, let's set up a configuration.\n\n## LoraConfig\n\nCreate a [`LoraConfig`] with the following parameters (or choose your own):\n\n```py\nfrom peft import LoraConfig\n\nconfig = LoraConfig(\n    r=16,\n    lora_alpha=8,\n    target_modules=[\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\"],\n    lora_dropout=0.05,\n    bias=\"none\",\n    task_type=\"CAUSAL_LM\"\n)\n```\n\nThen use the [`get_peft_model`] function to create a [`PeftModel`] from the quantized model and configuration.\n\n```py\nfrom peft import get_peft_model\n\nmodel = get_peft_model(model, config)\n```\n\nYou're all set for training with whichever training method you prefer!\n\n### LoftQ initialization\n\n[LoftQ](https://hf.co/papers/2310.08659) initializes LoRA weights such that the quantization error is minimized, and it can improve performance when training quantized models. To get started, follow [these instructions](https://github.com/huggingface/peft/tree/main/examples/loftq_finetuning).\n\nIn general, for LoftQ to work best, it is recommended to target as many layers with LoRA as possible, since those not targeted cannot have LoftQ applied. This means that passing `LoraConfig(..., target_modules=\"all-linear\")` will most likely give the best results. Also, you should use `nf4` as quant type in your quantization config when using 4bit quantization, i.e. `BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type=\"nf4\")`.\n\n### QLoRA-style training\n\nQLoRA adds trainable weights to all the linear layers in the transformer architecture."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: 4e5b791b-6862-462c-bdeb-a7eebc87375f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/developer_guides/quantization", "title": "Quantization", "tokens": 2385, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"4e5b791b-6862-462c-bdeb-a7eebc87375f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/quantization\", \"title\": \"Quantization\", \"tokens\": 2385, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"071f71de-9780-44e0-8fe6-0252113604f2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/quantization\", \"title\": \"Quantization\", \"tokens\": 2385, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"09bd4281c794e439948a282fe801dcbc6f8a70566ca7092c099c178e9bc863a3\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"be9091d1-d91f-494d-a0f7-65b34c25a7cb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/quantization\", \"title\": \"Quantization\", \"tokens\": 2385, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"445a1cab51ac499b4ad4f693540993d7e25bc8d0048ee4f0ae49020fa5ec65c0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ce357890-3d95-453c-bc0b-16dd70fc4f93\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"99037bae5f897518b172e4165a96a7c070c05c2017d8dcfc5c456869ec298aca\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2926, \"end_char_idx\": 5505, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "071f71de-9780-44e0-8fe6-0252113604f2", "doc_id": "071f71de-9780-44e0-8fe6-0252113604f2", "ref_doc_id": "071f71de-9780-44e0-8fe6-0252113604f2", "chroma:document": "## LoraConfig\n\nCreate a [`LoraConfig`] with the following parameters (or choose your own):\n\n```py\nfrom peft import LoraConfig\n\nconfig = LoraConfig(\n    r=16,\n    lora_alpha=8,\n    target_modules=[\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\"],\n    lora_dropout=0.05,\n    bias=\"none\",\n    task_type=\"CAUSAL_LM\"\n)\n```\n\nThen use the [`get_peft_model`] function to create a [`PeftModel`] from the quantized model and configuration.\n\n```py\nfrom peft import get_peft_model\n\nmodel = get_peft_model(model, config)\n```\n\nYou're all set for training with whichever training method you prefer!\n\n### LoftQ initialization\n\n[LoftQ](https://hf.co/papers/2310.08659) initializes LoRA weights such that the quantization error is minimized, and it can improve performance when training quantized models. To get started, follow [these instructions](https://github.com/huggingface/peft/tree/main/examples/loftq_finetuning).\n\nIn general, for LoftQ to work best, it is recommended to target as many layers with LoRA as possible, since those not targeted cannot have LoftQ applied. This means that passing `LoraConfig(..., target_modules=\"all-linear\")` will most likely give the best results. Also, you should use `nf4` as quant type in your quantization config when using 4bit quantization, i.e. `BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type=\"nf4\")`.\n\n### QLoRA-style training\n\nQLoRA adds trainable weights to all the linear layers in the transformer architecture. Since the attribute names for these linear layers can vary across architectures, set `target_modules` to `\"all-linear\"` to add LoRA to all the linear layers:\n\n```py\nconfig = LoraConfig(target_modules=\"all-linear\", ...)\n```\n\n## AQLM quantization\n\nAdditive Quantization of Language Models ([AQLM](https://arxiv.org/abs/2401.06118)) is a Large Language Models compression method. It quantizes multiple weights together and takes advantage of interdependencies between them. AQLM represents groups of 8-16 weights as a sum of multiple vector codes. This allows it to compress models down to as low as 2-bit with considerably low accuracy losses.\n\nSince the AQLM quantization process is computationally expensive, a use of prequantized models is recommended. A partial list of available models can be found in the official aqlm [repository](https://github.com/Vahe1994/AQLM).\n\nThe models support LoRA adapter tuning. To tune the quantized model you'll need to install the `aqlm` inference library: `pip install aqlm>=1.0.2`. Finetuned LoRA adapters shall be saved separately, as merging them with AQLM quantized weights is not possible."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: ce357890-3d95-453c-bc0b-16dd70fc4f93
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/developer_guides/quantization", "title": "Quantization", "tokens": 2385, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"ce357890-3d95-453c-bc0b-16dd70fc4f93\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/quantization\", \"title\": \"Quantization\", \"tokens\": 2385, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"071f71de-9780-44e0-8fe6-0252113604f2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/quantization\", \"title\": \"Quantization\", \"tokens\": 2385, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"09bd4281c794e439948a282fe801dcbc6f8a70566ca7092c099c178e9bc863a3\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4e5b791b-6862-462c-bdeb-a7eebc87375f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/quantization\", \"title\": \"Quantization\", \"tokens\": 2385, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"37da21ee4efb62b3791455f3ebf7b21187bad12ae63b3bd08e05ba8116c3e5da\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4823d966-db54-4844-9fd8-43e85abd2c0e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a1484d0ce013c38eea18650c0a8a1e475fda289eb566ca9c71cdf4b05e394352\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3977, \"end_char_idx\": 6558, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "071f71de-9780-44e0-8fe6-0252113604f2", "doc_id": "071f71de-9780-44e0-8fe6-0252113604f2", "ref_doc_id": "071f71de-9780-44e0-8fe6-0252113604f2", "chroma:document": "This means that passing `LoraConfig(..., target_modules=\"all-linear\")` will most likely give the best results. Also, you should use `nf4` as quant type in your quantization config when using 4bit quantization, i.e. `BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type=\"nf4\")`.\n\n### QLoRA-style training\n\nQLoRA adds trainable weights to all the linear layers in the transformer architecture. Since the attribute names for these linear layers can vary across architectures, set `target_modules` to `\"all-linear\"` to add LoRA to all the linear layers:\n\n```py\nconfig = LoraConfig(target_modules=\"all-linear\", ...)\n```\n\n## AQLM quantization\n\nAdditive Quantization of Language Models ([AQLM](https://arxiv.org/abs/2401.06118)) is a Large Language Models compression method. It quantizes multiple weights together and takes advantage of interdependencies between them. AQLM represents groups of 8-16 weights as a sum of multiple vector codes. This allows it to compress models down to as low as 2-bit with considerably low accuracy losses.\n\nSince the AQLM quantization process is computationally expensive, a use of prequantized models is recommended. A partial list of available models can be found in the official aqlm [repository](https://github.com/Vahe1994/AQLM).\n\nThe models support LoRA adapter tuning. To tune the quantized model you'll need to install the `aqlm` inference library: `pip install aqlm>=1.0.2`. Finetuned LoRA adapters shall be saved separately, as merging them with AQLM quantized weights is not possible.\n\n```py\nquantized_model = AutoModelForCausalLM.from_pretrained(\n    \"BlackSamorez/Mixtral-8x7b-AQLM-2Bit-1x16-hf-test-dispatch\",\n    torch_dtype=\"auto\", device_map=\"auto\", low_cpu_mem_usage=True,\n)\n\npeft_config = LoraConfig(...)\n\nquantized_model = get_peft_model(quantized_model, peft_config)\n```\n\nYou can refer to the [Google Colab](https://colab.research.google.com/drive/12GTp1FCj5_0SnnNQH18h_2XFh9vS_guX?usp=sharing) example for an overview of AQLM+LoRA finetuning.\n\n## EETQ quantization\n\nYou can also perform LoRA fine-tuning on EETQ quantized models. [EETQ](https://github.com/NetEase-FuXi/EETQ) package offers simple and efficient way to perform 8-bit quantization, which is claimed to be faster than the `LLM.int8()` algorithm. First, make sure that you have a transformers version that is compatible with EETQ (e.g. by installing it from latest pypi or from source).\n\n```py\nimport torch\nfrom transformers import EetqConfig\n\nconfig = EetqConfig(\"int8\")\n```\n\nPass the `config` to the [`~transformers.AutoModelForCausalLM.from_pretrained`] method."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: 4823d966-db54-4844-9fd8-43e85abd2c0e
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/developer_guides/quantization", "title": "Quantization", "tokens": 2385, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"4823d966-db54-4844-9fd8-43e85abd2c0e\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/quantization\", \"title\": \"Quantization\", \"tokens\": 2385, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"071f71de-9780-44e0-8fe6-0252113604f2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/quantization\", \"title\": \"Quantization\", \"tokens\": 2385, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"09bd4281c794e439948a282fe801dcbc6f8a70566ca7092c099c178e9bc863a3\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ce357890-3d95-453c-bc0b-16dd70fc4f93\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/quantization\", \"title\": \"Quantization\", \"tokens\": 2385, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"d17f4fbec2c38f7175b8b3f58c18d1d908a4e2a2ccac188256350737f9ca4544\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6e068a1f-aa08-4281-9b55-a4e9868a5d55\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c736cf5a7faaa1468d0d24785fca37dedf3d935faa0c61057279493eeee02c66\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5246, \"end_char_idx\": 7697, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "071f71de-9780-44e0-8fe6-0252113604f2", "doc_id": "071f71de-9780-44e0-8fe6-0252113604f2", "ref_doc_id": "071f71de-9780-44e0-8fe6-0252113604f2", "chroma:document": "The models support LoRA adapter tuning. To tune the quantized model you'll need to install the `aqlm` inference library: `pip install aqlm>=1.0.2`. Finetuned LoRA adapters shall be saved separately, as merging them with AQLM quantized weights is not possible.\n\n```py\nquantized_model = AutoModelForCausalLM.from_pretrained(\n    \"BlackSamorez/Mixtral-8x7b-AQLM-2Bit-1x16-hf-test-dispatch\",\n    torch_dtype=\"auto\", device_map=\"auto\", low_cpu_mem_usage=True,\n)\n\npeft_config = LoraConfig(...)\n\nquantized_model = get_peft_model(quantized_model, peft_config)\n```\n\nYou can refer to the [Google Colab](https://colab.research.google.com/drive/12GTp1FCj5_0SnnNQH18h_2XFh9vS_guX?usp=sharing) example for an overview of AQLM+LoRA finetuning.\n\n## EETQ quantization\n\nYou can also perform LoRA fine-tuning on EETQ quantized models. [EETQ](https://github.com/NetEase-FuXi/EETQ) package offers simple and efficient way to perform 8-bit quantization, which is claimed to be faster than the `LLM.int8()` algorithm. First, make sure that you have a transformers version that is compatible with EETQ (e.g. by installing it from latest pypi or from source).\n\n```py\nimport torch\nfrom transformers import EetqConfig\n\nconfig = EetqConfig(\"int8\")\n```\n\nPass the `config` to the [`~transformers.AutoModelForCausalLM.from_pretrained`] method.\n\n```py\nfrom transformers import AutoModelForCausalLM\n\nmodel = AutoModelForCausalLM.from_pretrained(\"mistralai/Mistral-7B-v0.1\", quantization_config=config)\n```\n\nand create a `LoraConfig` and pass it to `get_peft_model`:\n\n```py\nfrom peft import LoraConfig, get_peft_model\n\nconfig = LoraConfig(\n    r=16,\n    lora_alpha=8,\n    target_modules=[\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\"],\n    lora_dropout=0.05,\n    bias=\"none\",\n    task_type=\"CAUSAL_LM\"\n)\n\nmodel = get_peft_model(model, config)\n```\n\n## HQQ quantization\n\nThe models that is quantized using Half-Quadratic Quantization of Large Machine Learning Models ([HQQ](https://mobiusml.github.io/hqq_blog/)) support LoRA adapter tuning. To tune the quantized model, you'll need to install the `hqq` library with: `pip install hqq`.\n\n```python\nfrom hqq.engine.hf import HQQModelForCausalLM\n\nquantized_model = HQQModelForCausalLM.from_quantized(save_dir_or_hfhub, device='cuda')\npeft_config = LoraConfig(...)\nquantized_model = get_peft_model(quantized_model, peft_config)\n```\n\nOr using transformers version that is compatible with HQQ (e.g. by installing it from latest pypi or from source)."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: 6e068a1f-aa08-4281-9b55-a4e9868a5d55
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/developer_guides/quantization", "title": "Quantization", "tokens": 2385, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"6e068a1f-aa08-4281-9b55-a4e9868a5d55\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/quantization\", \"title\": \"Quantization\", \"tokens\": 2385, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"071f71de-9780-44e0-8fe6-0252113604f2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/quantization\", \"title\": \"Quantization\", \"tokens\": 2385, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"09bd4281c794e439948a282fe801dcbc6f8a70566ca7092c099c178e9bc863a3\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4823d966-db54-4844-9fd8-43e85abd2c0e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/quantization\", \"title\": \"Quantization\", \"tokens\": 2385, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"b2f2cafc6c9740dd722f2e61025f504b19a35267411294b0bf339484bb14853b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6330, \"end_char_idx\": 8534, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "071f71de-9780-44e0-8fe6-0252113604f2", "doc_id": "071f71de-9780-44e0-8fe6-0252113604f2", "ref_doc_id": "071f71de-9780-44e0-8fe6-0252113604f2", "chroma:document": "by installing it from latest pypi or from source).\n\n```py\nimport torch\nfrom transformers import EetqConfig\n\nconfig = EetqConfig(\"int8\")\n```\n\nPass the `config` to the [`~transformers.AutoModelForCausalLM.from_pretrained`] method.\n\n```py\nfrom transformers import AutoModelForCausalLM\n\nmodel = AutoModelForCausalLM.from_pretrained(\"mistralai/Mistral-7B-v0.1\", quantization_config=config)\n```\n\nand create a `LoraConfig` and pass it to `get_peft_model`:\n\n```py\nfrom peft import LoraConfig, get_peft_model\n\nconfig = LoraConfig(\n    r=16,\n    lora_alpha=8,\n    target_modules=[\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\"],\n    lora_dropout=0.05,\n    bias=\"none\",\n    task_type=\"CAUSAL_LM\"\n)\n\nmodel = get_peft_model(model, config)\n```\n\n## HQQ quantization\n\nThe models that is quantized using Half-Quadratic Quantization of Large Machine Learning Models ([HQQ](https://mobiusml.github.io/hqq_blog/)) support LoRA adapter tuning. To tune the quantized model, you'll need to install the `hqq` library with: `pip install hqq`.\n\n```python\nfrom hqq.engine.hf import HQQModelForCausalLM\n\nquantized_model = HQQModelForCausalLM.from_quantized(save_dir_or_hfhub, device='cuda')\npeft_config = LoraConfig(...)\nquantized_model = get_peft_model(quantized_model, peft_config)\n```\n\nOr using transformers version that is compatible with HQQ (e.g. by installing it from latest pypi or from source).\n\n```python\nfrom transformers import HqqConfig, AutoModelForCausalLM\n\nquant_config = HqqConfig(nbits=4, group_size=64)\nquantized_model = AutoModelForCausalLM.from_pretrained(save_dir_or_hfhub, device_map=device_map, quantization_config=quant_config)\npeft_config = LoraConfig(...)\nquantized_model = get_peft_model(quantized_model, peft_config)\n```\n\n## Next steps\n\nIf you're interested in learning more about quantization, the following may be helpful:\n\n* Learn more about details about QLoRA and check out some benchmarks on its impact in the [Making LLMs even more accessible with bitsandbytes, 4-bit quantization and QLoRA](https://huggingface.co/blog/4bit-transformers-bitsandbytes) blog post.\n* Read more about different quantization schemes in the Transformers [Quantization](https://hf.co/docs/transformers/main/quantization) guide."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: 417d16a9-1c04-4183-8344-ae66e2fdb739
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/developer_guides/model_merging", "title": "Model merging", "tokens": 2033, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"417d16a9-1c04-4183-8344-ae66e2fdb739\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/model_merging\", \"title\": \"Model merging\", \"tokens\": 2033, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e75cdd83-c302-4f98-aa23-b52a23ddf01c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/model_merging\", \"title\": \"Model merging\", \"tokens\": 2033, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"037d7dabf7d6c167c55600c9cf6f9be0e655a76409c0718de353a56c34df052b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"772f443e-af1b-445e-8d0d-86870ecb91c8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3bcebc537ff6e9be53d47053aa2124f98f7bdc230ce4d9af39602fbf09d10cce\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2909, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e75cdd83-c302-4f98-aa23-b52a23ddf01c", "doc_id": "e75cdd83-c302-4f98-aa23-b52a23ddf01c", "ref_doc_id": "e75cdd83-c302-4f98-aa23-b52a23ddf01c", "chroma:document": "# Model merging\n\nTraining a model for each task can be costly, take up storage space, and the models aren't able to learn new information to improve their performance. Multitask learning can overcome some of these limitations by training a model to learn several tasks, but it is expensive to train and designing a dataset for it is challenging. *Model merging* offers a solution to these challenges by combining multiple pretrained models into one model, giving it the combined abilities of each individual model without any additional training.\n\nPEFT provides several methods for merging models like a linear or SVD combination. This guide focuses on two methods that are more efficient for merging LoRA adapters by eliminating redundant parameters:\n\n* [TIES](https://hf.co/papers/2306.01708) - TrIm, Elect, and Merge (TIES) is a three-step method for merging models. First, redundant parameters are trimmed, then conflicting signs are resolved into an aggregated vector, and finally the parameters whose signs are the same as the aggregate sign are averaged. This method takes into account that some values (redundant and sign disagreement) can degrade performance in the merged model.\n* [DARE](https://hf.co/papers/2311.03099) - Drop And REscale is a method that can be used to prepare for other model merging methods like TIES. It works by randomly dropping parameters according to a drop rate and rescaling the remaining parameters. This helps to reduce the number of redundant and potentially interfering parameters among multiple models.\n\nModels are merged with the [`~LoraModel.add_weighted_adapter`] method, and the specific model merging method is specified in the `combination_type` parameter.\n\n## Merge method\n\nWith TIES and DARE, merging is enabled by setting `combination_type` and `density` to a value of the weights to keep from the individual models. For example, let's merge three finetuned [TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T](https://huggingface.co/TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T) models: [tinyllama_lora_nobots](https://huggingface.co/smangrul/tinyllama_lora_norobots), [tinyllama_lora_sql](https://huggingface.co/smangrul/tinyllama_lora_sql), and [tinyllama_lora_adcopy](https://huggingface.co/smangrul/tinyllama_lora_adcopy).\n\n<Tip warninig={true}>\n\nWhen you're attempting to merge fully trained models with TIES, you should be aware of any special tokens each model may have added to the embedding layer which are not a part of the original checkpoint's vocabulary. This may cause an issue because each model may have added a special token to the same embedding position. If this is the case, you should use the [`~transformers.PreTrainedModel.resize_token_embeddings`] method to avoid merging the special tokens at the same embedding index.\n\n<br>\n\nThis shouldn't be an issue if you're only merging LoRA adapters trained from the same base model."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: 772f443e-af1b-445e-8d0d-86870ecb91c8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/developer_guides/model_merging", "title": "Model merging", "tokens": 2033, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"772f443e-af1b-445e-8d0d-86870ecb91c8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/model_merging\", \"title\": \"Model merging\", \"tokens\": 2033, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e75cdd83-c302-4f98-aa23-b52a23ddf01c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/model_merging\", \"title\": \"Model merging\", \"tokens\": 2033, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"037d7dabf7d6c167c55600c9cf6f9be0e655a76409c0718de353a56c34df052b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"417d16a9-1c04-4183-8344-ae66e2fdb739\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/model_merging\", \"title\": \"Model merging\", \"tokens\": 2033, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"5341ce8a43d6c3dcc4de1447cf247a2d6c75eb8ae063d841d7a3013b9f565bb0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2c21a00a-6150-4278-9a3d-73096cfc85f2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ff1be3560bc7817c182d63de6ddd7a314785a4cec6f84285b9d0ead527dd0fac\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1333, \"end_char_idx\": 4049, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e75cdd83-c302-4f98-aa23-b52a23ddf01c", "doc_id": "e75cdd83-c302-4f98-aa23-b52a23ddf01c", "ref_doc_id": "e75cdd83-c302-4f98-aa23-b52a23ddf01c", "chroma:document": "It works by randomly dropping parameters according to a drop rate and rescaling the remaining parameters. This helps to reduce the number of redundant and potentially interfering parameters among multiple models.\n\nModels are merged with the [`~LoraModel.add_weighted_adapter`] method, and the specific model merging method is specified in the `combination_type` parameter.\n\n## Merge method\n\nWith TIES and DARE, merging is enabled by setting `combination_type` and `density` to a value of the weights to keep from the individual models. For example, let's merge three finetuned [TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T](https://huggingface.co/TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T) models: [tinyllama_lora_nobots](https://huggingface.co/smangrul/tinyllama_lora_norobots), [tinyllama_lora_sql](https://huggingface.co/smangrul/tinyllama_lora_sql), and [tinyllama_lora_adcopy](https://huggingface.co/smangrul/tinyllama_lora_adcopy).\n\n<Tip warninig={true}>\n\nWhen you're attempting to merge fully trained models with TIES, you should be aware of any special tokens each model may have added to the embedding layer which are not a part of the original checkpoint's vocabulary. This may cause an issue because each model may have added a special token to the same embedding position. If this is the case, you should use the [`~transformers.PreTrainedModel.resize_token_embeddings`] method to avoid merging the special tokens at the same embedding index.\n\n<br>\n\nThis shouldn't be an issue if you're only merging LoRA adapters trained from the same base model.\n\n</Tip>\n\nLoad a base model and can use the [`~PeftModel.load_adapter`] method to load and assign each adapter a name:\n\n```py\nfrom peft import PeftConfig, PeftModel\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nimport torch\n\nconfig = PeftConfig.from_pretrained(\"smangrul/tinyllama_lora_norobots\")\nmodel = AutoModelForCausalLM.from_pretrained(config.base_model_name_or_path, load_in_4bit=True, device_map=\"auto\").eval()\ntokenizer = AutoTokenizer.from_pretrained(\"smangrul/tinyllama_lora_norobots\")\n\nmodel = PeftModel.from_pretrained(model, \"smangrul/tinyllama_lora_norobots\", adapter_name=\"norobots\")\n_ = model.load_adapter(\"smangrul/tinyllama_lora_sql\", adapter_name=\"sql\")\n_ = model.load_adapter(\"smangrul/tinyllama_lora_adcopy\", adapter_name=\"adcopy\")\n```\n\nSet the adapters, weights, `adapter_name`, `combination_type`, and `density` with the [`~LoraModel.add_weighted_adapter`] method.\n\n<hfoptions id=\"merge-method\">\n<hfoption id=\"TIES\">\n\nWeight values greater than `1.0` typically produce better results because they preserve the correct scale. A good default starting value for the weights is to set all values to `1.0`."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: 2c21a00a-6150-4278-9a3d-73096cfc85f2
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/developer_guides/model_merging", "title": "Model merging", "tokens": 2033, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"2c21a00a-6150-4278-9a3d-73096cfc85f2\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/model_merging\", \"title\": \"Model merging\", \"tokens\": 2033, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e75cdd83-c302-4f98-aa23-b52a23ddf01c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/model_merging\", \"title\": \"Model merging\", \"tokens\": 2033, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"037d7dabf7d6c167c55600c9cf6f9be0e655a76409c0718de353a56c34df052b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"772f443e-af1b-445e-8d0d-86870ecb91c8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/model_merging\", \"title\": \"Model merging\", \"tokens\": 2033, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"4d37961ad26c7c14eba2c095bba16d00437480c051178e7aec8d70e3da076cf2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"08401011-109d-40c6-b55f-46fcd80228a8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"43a515ea3df3b8b92306d651d7a09e5d53c1ef792a4347552b44712293104117\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2529, \"end_char_idx\": 4912, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e75cdd83-c302-4f98-aa23-b52a23ddf01c", "doc_id": "e75cdd83-c302-4f98-aa23-b52a23ddf01c", "ref_doc_id": "e75cdd83-c302-4f98-aa23-b52a23ddf01c", "chroma:document": "This may cause an issue because each model may have added a special token to the same embedding position. If this is the case, you should use the [`~transformers.PreTrainedModel.resize_token_embeddings`] method to avoid merging the special tokens at the same embedding index.\n\n<br>\n\nThis shouldn't be an issue if you're only merging LoRA adapters trained from the same base model.\n\n</Tip>\n\nLoad a base model and can use the [`~PeftModel.load_adapter`] method to load and assign each adapter a name:\n\n```py\nfrom peft import PeftConfig, PeftModel\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nimport torch\n\nconfig = PeftConfig.from_pretrained(\"smangrul/tinyllama_lora_norobots\")\nmodel = AutoModelForCausalLM.from_pretrained(config.base_model_name_or_path, load_in_4bit=True, device_map=\"auto\").eval()\ntokenizer = AutoTokenizer.from_pretrained(\"smangrul/tinyllama_lora_norobots\")\n\nmodel = PeftModel.from_pretrained(model, \"smangrul/tinyllama_lora_norobots\", adapter_name=\"norobots\")\n_ = model.load_adapter(\"smangrul/tinyllama_lora_sql\", adapter_name=\"sql\")\n_ = model.load_adapter(\"smangrul/tinyllama_lora_adcopy\", adapter_name=\"adcopy\")\n```\n\nSet the adapters, weights, `adapter_name`, `combination_type`, and `density` with the [`~LoraModel.add_weighted_adapter`] method.\n\n<hfoptions id=\"merge-method\">\n<hfoption id=\"TIES\">\n\nWeight values greater than `1.0` typically produce better results because they preserve the correct scale. A good default starting value for the weights is to set all values to `1.0`.\n\n```py\nadapters = [\"norobots\", \"adcopy\", \"sql\"]\nweights = [2.0, 1.0, 1.0]\nadapter_name = \"merge\"\ndensity = 0.2\nmodel.add_weighted_adapter(adapters, weights, adapter_name, combination_type=\"ties\", density=density)\n```\n\n</hfoption>\n<hfoption id=\"DARE\">\n\n```py\nadapters = [\"norobots\", \"adcopy\", \"sql\"]\nweights = [2.0, 0.3, 0.7]\nadapter_name = \"merge\"\ndensity = 0.2\nmodel.add_weighted_adapter(adapters, weights, adapter_name, combination_type=\"dare_ties\", density=density)\n```\n\n</hfoption>\n</hfoptions>\n\nSet the newly merged model as the active model with the [`~LoraModel.set_adapter`] method.\n\n```py\nmodel.set_adapter(\"merge\")\n```\n\nNow you can use the merged model as an instruction-tuned model to write ad copy or SQL queries!\n\n<hfoptions id=\"ties\">\n<hfoption id=\"instruct\">\n\n```py\nmessages = [\n    {\"role\": \"user\", \"content\": \"Write an essay about Generative AI."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: 08401011-109d-40c6-b55f-46fcd80228a8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/developer_guides/model_merging", "title": "Model merging", "tokens": 2033, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"08401011-109d-40c6-b55f-46fcd80228a8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/model_merging\", \"title\": \"Model merging\", \"tokens\": 2033, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e75cdd83-c302-4f98-aa23-b52a23ddf01c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/model_merging\", \"title\": \"Model merging\", \"tokens\": 2033, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"037d7dabf7d6c167c55600c9cf6f9be0e655a76409c0718de353a56c34df052b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2c21a00a-6150-4278-9a3d-73096cfc85f2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/model_merging\", \"title\": \"Model merging\", \"tokens\": 2033, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"f9caf5b50362604a325b3cf8d63f816b5beaf35cbed9699d85fa062b98b19970\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8d319f77-5858-4c6e-97bf-c9193515cae3\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"be6fd733379a609f5a96dc091a2fe36e56c82e29baa2269fbc952ed427bec66a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3814, \"end_char_idx\": 6559, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e75cdd83-c302-4f98-aa23-b52a23ddf01c", "doc_id": "e75cdd83-c302-4f98-aa23-b52a23ddf01c", "ref_doc_id": "e75cdd83-c302-4f98-aa23-b52a23ddf01c", "chroma:document": "<hfoptions id=\"merge-method\">\n<hfoption id=\"TIES\">\n\nWeight values greater than `1.0` typically produce better results because they preserve the correct scale. A good default starting value for the weights is to set all values to `1.0`.\n\n```py\nadapters = [\"norobots\", \"adcopy\", \"sql\"]\nweights = [2.0, 1.0, 1.0]\nadapter_name = \"merge\"\ndensity = 0.2\nmodel.add_weighted_adapter(adapters, weights, adapter_name, combination_type=\"ties\", density=density)\n```\n\n</hfoption>\n<hfoption id=\"DARE\">\n\n```py\nadapters = [\"norobots\", \"adcopy\", \"sql\"]\nweights = [2.0, 0.3, 0.7]\nadapter_name = \"merge\"\ndensity = 0.2\nmodel.add_weighted_adapter(adapters, weights, adapter_name, combination_type=\"dare_ties\", density=density)\n```\n\n</hfoption>\n</hfoptions>\n\nSet the newly merged model as the active model with the [`~LoraModel.set_adapter`] method.\n\n```py\nmodel.set_adapter(\"merge\")\n```\n\nNow you can use the merged model as an instruction-tuned model to write ad copy or SQL queries!\n\n<hfoptions id=\"ties\">\n<hfoption id=\"instruct\">\n\n```py\nmessages = [\n    {\"role\": \"user\", \"content\": \"Write an essay about Generative AI.\"},\n]\ntext = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)\ninputs = tokenizer(text, return_tensors=\"pt\")\ninputs = {k: v.to(\"cuda\") for k, v in inputs.items()}\noutputs = model.generate(**inputs, max_new_tokens=256, do_sample=True, top_p=0.95, temperature=0.2, repetition_penalty=1.2, eos_token_id=tokenizer.eos_token_id)\nprint(tokenizer.decode(outputs[0]))\n```\n\n</hfoption>\n<hfoption id=\"ad copy\">\n\n```py\nmessages = [\n    {\"role\": \"system\", \"content\": \"Create a text ad given the following product and description.\"},\n    {\"role\": \"user\", \"content\": \"Product: Sony PS5 PlayStation Console\\nDescription: The PS5 console unleashes new gaming possibilities that you never anticipated.\"},\n]\ntext = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)\ninputs = tokenizer(text, return_tensors=\"pt\")\ninputs = {k: v.to(\"cuda\") for k, v in inputs.items()}\noutputs = model.generate(**inputs, max_new_tokens=128, do_sample=True, top_p=0.95, temperature=0.2, repetition_penalty=1.2, eos_token_id=tokenizer.eos_token_id)\nprint(tokenizer.decode(outputs[0]))\n```\n\n</hfoption>\n<hfoption id=\"SQL\">\n\n```py\ntext = \"\"\"Table: 2-11365528-2\nColumns: ['Team', 'Head Coach', 'President', 'Home Ground', 'Location']\nNatural Query: Who is the Head Coach of the team whose President is Mario Volarevic?\nSQL Query:\"\"\"\n\ninputs = tokenizer(text, return_tensors=\"pt\")\ninputs = {k: v.to(\"cuda\") for k, v in inputs.items()}\noutputs = model.generate(**inputs, max_new_tokens=64, repetition_penalty=1.1, eos_token_id=tokenizer(\"</s>\").input_ids[-1])\nprint(tokenizer.decode(outputs[0]))\n```\n\n</hfoption>\n</hfoptions>"}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: 8d319f77-5858-4c6e-97bf-c9193515cae3
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/developer_guides/model_merging", "title": "Model merging", "tokens": 2033, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"8d319f77-5858-4c6e-97bf-c9193515cae3\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/model_merging\", \"title\": \"Model merging\", \"tokens\": 2033, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e75cdd83-c302-4f98-aa23-b52a23ddf01c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/model_merging\", \"title\": \"Model merging\", \"tokens\": 2033, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"037d7dabf7d6c167c55600c9cf6f9be0e655a76409c0718de353a56c34df052b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"08401011-109d-40c6-b55f-46fcd80228a8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/developer_guides/model_merging\", \"title\": \"Model merging\", \"tokens\": 2033, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"5f7c6e2b3eb1cda91c84c30d9148f9e979d6e102453a2c458f864d5c4440df0a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5463, \"end_char_idx\": 7375, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e75cdd83-c302-4f98-aa23-b52a23ddf01c", "doc_id": "e75cdd83-c302-4f98-aa23-b52a23ddf01c", "ref_doc_id": "e75cdd83-c302-4f98-aa23-b52a23ddf01c", "chroma:document": "\"},\n    {\"role\": \"user\", \"content\": \"Product: Sony PS5 PlayStation Console\\nDescription: The PS5 console unleashes new gaming possibilities that you never anticipated.\"},\n]\ntext = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)\ninputs = tokenizer(text, return_tensors=\"pt\")\ninputs = {k: v.to(\"cuda\") for k, v in inputs.items()}\noutputs = model.generate(**inputs, max_new_tokens=128, do_sample=True, top_p=0.95, temperature=0.2, repetition_penalty=1.2, eos_token_id=tokenizer.eos_token_id)\nprint(tokenizer.decode(outputs[0]))\n```\n\n</hfoption>\n<hfoption id=\"SQL\">\n\n```py\ntext = \"\"\"Table: 2-11365528-2\nColumns: ['Team', 'Head Coach', 'President', 'Home Ground', 'Location']\nNatural Query: Who is the Head Coach of the team whose President is Mario Volarevic?\nSQL Query:\"\"\"\n\ninputs = tokenizer(text, return_tensors=\"pt\")\ninputs = {k: v.to(\"cuda\") for k, v in inputs.items()}\noutputs = model.generate(**inputs, max_new_tokens=64, repetition_penalty=1.1, eos_token_id=tokenizer(\"</s>\").input_ids[-1])\nprint(tokenizer.decode(outputs[0]))\n```\n\n</hfoption>\n</hfoptions>\n\n\n## Merging (IA)\u00b3 Models\nThe (IA)\u00b3 models facilitate linear merging of adapters. To merge adapters in an (IA)\u00b3 model, utilize the `add_weighted_adapter` method from the `IA3Model` class. This method is analogous to the `add_weighted_adapter` method used in `LoraModel`, with the key difference being the absence of the `combination_type` parameter. For example, to merge three (IA)\u00b3 adapters into a PEFT model, you would proceed as follows:\n\n```py\nadapters = [\"adapter1\", \"adapter2\", \"adapter3\"]\nweights = [0.4, 0.3, 0.3]\nadapter_name = \"merge\"\nmodel.add_weighted_adapter(adapters, weights, adapter_name)\n```\n\nIt is recommended that the weights sum to 1.0 to preserve the scale of the model. The merged model can then be set as the active model using the `set_adapter` method:\n\n```py\nmodel.set_adapter(\"merge\")\n```"}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: 4646d284-2d3a-4d69-a26b-6bead84b5aca
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/task_guides/ia3", "title": "IA3", "tokens": 2278, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"4646d284-2d3a-4d69-a26b-6bead84b5aca\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/ia3\", \"title\": \"IA3\", \"tokens\": 2278, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"abaf329e-104c-41c1-9818-3eacccc542f6\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/ia3\", \"title\": \"IA3\", \"tokens\": 2278, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"c1393fd481102f1fef41ff3d4b2f88fbdf1a56a876eaf96f6918e3274864df49\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"67146300-aaf9-4072-8a8e-0f5da54d157a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c97a78c4dd7f0a68dc6810255cf39c7bdb5fb4b8277a973d492d3ae4b75ece5d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2447, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "abaf329e-104c-41c1-9818-3eacccc542f6", "doc_id": "abaf329e-104c-41c1-9818-3eacccc542f6", "ref_doc_id": "abaf329e-104c-41c1-9818-3eacccc542f6", "chroma:document": "# IA3\n\n[IA3](../conceptual_guides/ia3) multiplies the model's activations (the keys and values in the self-attention and encoder-decoder attention blocks, and the intermediate activation of the position-wise feedforward network) by three learned vectors. This PEFT method introduces an even smaller number of trainable parameters than LoRA which introduces weight matrices instead of vectors. The original model's parameters are kept frozen and only these vectors are updated. As a result, it is faster, cheaper and more efficient to finetune for a new downstream task.\n\nThis guide will show you how to train a sequence-to-sequence model with IA3 to *generate a sentiment* given some financial news.\n\n<Tip>\n\nSome familiarity with the general process of training a sequence-to-sequence would be really helpful and allow you to focus on how to apply IA3. If you\u2019re new, we recommend taking a look at the [Translation](https://huggingface.co/docs/transformers/tasks/translation) and [Summarization](https://huggingface.co/docs/transformers/tasks/summarization) guides first from the Transformers documentation. When you\u2019re ready, come back and see how easy it is to drop PEFT in to your training!\n\n</Tip>\n\n## Dataset\n\nYou'll use the sentences_allagree subset of the [financial_phrasebank](https://huggingface.co/datasets/financial_phrasebank) dataset. This subset contains financial news with 100% annotator agreement on the sentiment label. Take a look at the [dataset viewer](https://huggingface.co/datasets/financial_phrasebank/viewer/sentences_allagree) for a better idea of the data and sentences you'll be working with.\n\nLoad the dataset with the [`~datasets.load_dataset`] function. This subset of the dataset only contains a train split, so use the [`~datasets.train_test_split`] function to create a train and validation split. Create a new `text_label` column so it is easier to understand what the `label` values `0`, `1`, and `2` mean.\n\n```py\nfrom datasets import load_dataset\n\nds = load_dataset(\"financial_phrasebank\", \"sentences_allagree\")\nds = ds[\"train\"].train_test_split(test_size=0.1)\nds[\"validation\"] = ds[\"test\"]\ndel ds[\"test\"]\n\nclasses = ds[\"train\"].features[\"label\"].names\nds = ds.map(\n    lambda x: {\"text_label\": [classes[label] for label in x[\"label\"]]},\n    batched=True,\n    num_proc=1,\n)\n\nds[\"train\"][0]\n{'sentence': 'It will be operated by Nokia , and supported by its Nokia NetAct network and service management system ."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: 67146300-aaf9-4072-8a8e-0f5da54d157a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/task_guides/ia3", "title": "IA3", "tokens": 2278, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"67146300-aaf9-4072-8a8e-0f5da54d157a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/ia3\", \"title\": \"IA3\", \"tokens\": 2278, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"abaf329e-104c-41c1-9818-3eacccc542f6\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/ia3\", \"title\": \"IA3\", \"tokens\": 2278, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"c1393fd481102f1fef41ff3d4b2f88fbdf1a56a876eaf96f6918e3274864df49\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4646d284-2d3a-4d69-a26b-6bead84b5aca\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/ia3\", \"title\": \"IA3\", \"tokens\": 2278, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"fd14fc3a1fd3f0c2415c34d0a591c06d8721aa694212f61b461246e21dd0de05\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4dbcea7f-8516-4b0e-9686-e09531f545d8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"17d5a057dc0ffa8d111096c7dc30b6c5d5f2334282e86560c2748bd339b71b2c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 853, \"end_char_idx\": 3982, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "abaf329e-104c-41c1-9818-3eacccc542f6", "doc_id": "abaf329e-104c-41c1-9818-3eacccc542f6", "ref_doc_id": "abaf329e-104c-41c1-9818-3eacccc542f6", "chroma:document": "If you\u2019re new, we recommend taking a look at the [Translation](https://huggingface.co/docs/transformers/tasks/translation) and [Summarization](https://huggingface.co/docs/transformers/tasks/summarization) guides first from the Transformers documentation. When you\u2019re ready, come back and see how easy it is to drop PEFT in to your training!\n\n</Tip>\n\n## Dataset\n\nYou'll use the sentences_allagree subset of the [financial_phrasebank](https://huggingface.co/datasets/financial_phrasebank) dataset. This subset contains financial news with 100% annotator agreement on the sentiment label. Take a look at the [dataset viewer](https://huggingface.co/datasets/financial_phrasebank/viewer/sentences_allagree) for a better idea of the data and sentences you'll be working with.\n\nLoad the dataset with the [`~datasets.load_dataset`] function. This subset of the dataset only contains a train split, so use the [`~datasets.train_test_split`] function to create a train and validation split. Create a new `text_label` column so it is easier to understand what the `label` values `0`, `1`, and `2` mean.\n\n```py\nfrom datasets import load_dataset\n\nds = load_dataset(\"financial_phrasebank\", \"sentences_allagree\")\nds = ds[\"train\"].train_test_split(test_size=0.1)\nds[\"validation\"] = ds[\"test\"]\ndel ds[\"test\"]\n\nclasses = ds[\"train\"].features[\"label\"].names\nds = ds.map(\n    lambda x: {\"text_label\": [classes[label] for label in x[\"label\"]]},\n    batched=True,\n    num_proc=1,\n)\n\nds[\"train\"][0]\n{'sentence': 'It will be operated by Nokia , and supported by its Nokia NetAct network and service management system .',\n 'label': 1,\n 'text_label': 'neutral'}\n```\n\nLoad a tokenizer and create a preprocessing function that:\n\n1. tokenizes the inputs, pads and truncates the sequence to the `max_length`\n2. apply the same tokenizer to the labels but with a shorter `max_length` that corresponds to the label\n3. mask the padding tokens\n\n```py\nfrom transformers import AutoTokenizer\n\ntext_column = \"sentence\"\nlabel_column = \"text_label\"\nmax_length = 128\n\ntokenizer = AutoTokenizer.from_pretrained(\"bigscience/mt0-large\")\n\ndef preprocess_function(examples):\n    inputs = examples[text_column]\n    targets = examples[label_column]\n    model_inputs = tokenizer(inputs, max_length=max_length, padding=\"max_length\", truncation=True, return_tensors=\"pt\")\n    labels = tokenizer(targets, max_length=3, padding=\"max_length\", truncation=True, return_tensors=\"pt\")\n    labels = labels[\"input_ids\"]\n    labels[labels == tokenizer.pad_token_id] = -100\n    model_inputs[\"labels\"] = labels\n    return model_inputs\n```\n\nUse the [`~datasets.Dataset.map`] function to apply the preprocessing function to the entire dataset.\n\n```py\nprocessed_ds = ds.map(\n    preprocess_function,\n    batched=True,\n    num_proc=1,\n    remove_columns=ds[\"train\"].column_names,\n    load_from_cache_file=False,\n    desc=\"Running tokenizer on dataset\",\n)\n```\n\nCreate a training and evaluation [`DataLoader`](https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader), and set `pin_memory=True` to speed up data transfer to the GPU during training if your dataset samples are on a CPU."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: 4dbcea7f-8516-4b0e-9686-e09531f545d8
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/task_guides/ia3", "title": "IA3", "tokens": 2278, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"4dbcea7f-8516-4b0e-9686-e09531f545d8\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/ia3\", \"title\": \"IA3\", \"tokens\": 2278, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"abaf329e-104c-41c1-9818-3eacccc542f6\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/ia3\", \"title\": \"IA3\", \"tokens\": 2278, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"c1393fd481102f1fef41ff3d4b2f88fbdf1a56a876eaf96f6918e3274864df49\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"67146300-aaf9-4072-8a8e-0f5da54d157a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/ia3\", \"title\": \"IA3\", \"tokens\": 2278, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"f4c4202dd0b073daeb50d424f39f9a375fc43e2f9ef36b9acc8a17aa9c6cf297\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f01c968e-3d7a-42a2-b701-fcfab7f2b713\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ccdc10876bddff76b0e2290efcbfa77ccf8e7fcfb750a1f96ab1839f7e18f72b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2447, \"end_char_idx\": 5467, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "abaf329e-104c-41c1-9818-3eacccc542f6", "doc_id": "abaf329e-104c-41c1-9818-3eacccc542f6", "ref_doc_id": "abaf329e-104c-41c1-9818-3eacccc542f6", "chroma:document": "',\n 'label': 1,\n 'text_label': 'neutral'}\n```\n\nLoad a tokenizer and create a preprocessing function that:\n\n1. tokenizes the inputs, pads and truncates the sequence to the `max_length`\n2. apply the same tokenizer to the labels but with a shorter `max_length` that corresponds to the label\n3. mask the padding tokens\n\n```py\nfrom transformers import AutoTokenizer\n\ntext_column = \"sentence\"\nlabel_column = \"text_label\"\nmax_length = 128\n\ntokenizer = AutoTokenizer.from_pretrained(\"bigscience/mt0-large\")\n\ndef preprocess_function(examples):\n    inputs = examples[text_column]\n    targets = examples[label_column]\n    model_inputs = tokenizer(inputs, max_length=max_length, padding=\"max_length\", truncation=True, return_tensors=\"pt\")\n    labels = tokenizer(targets, max_length=3, padding=\"max_length\", truncation=True, return_tensors=\"pt\")\n    labels = labels[\"input_ids\"]\n    labels[labels == tokenizer.pad_token_id] = -100\n    model_inputs[\"labels\"] = labels\n    return model_inputs\n```\n\nUse the [`~datasets.Dataset.map`] function to apply the preprocessing function to the entire dataset.\n\n```py\nprocessed_ds = ds.map(\n    preprocess_function,\n    batched=True,\n    num_proc=1,\n    remove_columns=ds[\"train\"].column_names,\n    load_from_cache_file=False,\n    desc=\"Running tokenizer on dataset\",\n)\n```\n\nCreate a training and evaluation [`DataLoader`](https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader), and set `pin_memory=True` to speed up data transfer to the GPU during training if your dataset samples are on a CPU.\n\n```py\nfrom torch.utils.data import DataLoader\nfrom transformers import default_data_collator\n\ntrain_ds = processed_ds[\"train\"]\neval_ds = processed_ds[\"validation\"]\n\nbatch_size = 8\n\ntrain_dataloader = DataLoader(\n    train_ds, shuffle=True, collate_fn=default_data_collator, batch_size=batch_size, pin_memory=True\n)\neval_dataloader = DataLoader(eval_ds, collate_fn=default_data_collator, batch_size=batch_size, pin_memory=True)\n```\n\n## Model\n\nNow you can load a pretrained model to use as the base model for IA3. This guide uses the [bigscience/mt0-large](https://huggingface.co/bigscience/mt0-large) model, but you can use any sequence-to-sequence model you like.\n\n```py\nfrom transformers import AutoModelForSeq2SeqLM\n\nmodel = AutoModelForSeq2SeqLM.from_pretrained(\"bigscience/mt0-large\")\n```\n\n### PEFT configuration and model\n\nAll PEFT methods need a configuration that contains and specifies all the parameters for how the PEFT method should be applied. Create an [`IA3Config`] with the task type and set the inference mode to `False`. You can find additional parameters for this configuration in the [API reference](../package_reference/ia3#ia3config).\n\n<Tip>\n\nCall the [`~PeftModel.print_trainable_parameters`] method to compare the number of trainable parameters of [`PeftModel`] versus the number of parameters in the base model!\n\n</Tip>\n\nOnce the configuration is setup, pass it to the [`get_peft_model`] function along with the base model to create a trainable [`PeftModel`]."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: f01c968e-3d7a-42a2-b701-fcfab7f2b713
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/task_guides/ia3", "title": "IA3", "tokens": 2278, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"f01c968e-3d7a-42a2-b701-fcfab7f2b713\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/ia3\", \"title\": \"IA3\", \"tokens\": 2278, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"abaf329e-104c-41c1-9818-3eacccc542f6\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/ia3\", \"title\": \"IA3\", \"tokens\": 2278, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"c1393fd481102f1fef41ff3d4b2f88fbdf1a56a876eaf96f6918e3274864df49\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4dbcea7f-8516-4b0e-9686-e09531f545d8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/ia3\", \"title\": \"IA3\", \"tokens\": 2278, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"f34c366452687774aeb6506db7f9c5c272541ba1c2c43b8ba0c3b5f0aab9f0ac\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"126a6414-bfdd-43bb-b083-8184e526d64f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"684f3721f5f000c72e93abdb2c64e45c0e61c5bdd37bddd0ca0cced01968e6d4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3984, \"end_char_idx\": 6239, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "abaf329e-104c-41c1-9818-3eacccc542f6", "doc_id": "abaf329e-104c-41c1-9818-3eacccc542f6", "ref_doc_id": "abaf329e-104c-41c1-9818-3eacccc542f6", "chroma:document": "```py\nfrom torch.utils.data import DataLoader\nfrom transformers import default_data_collator\n\ntrain_ds = processed_ds[\"train\"]\neval_ds = processed_ds[\"validation\"]\n\nbatch_size = 8\n\ntrain_dataloader = DataLoader(\n    train_ds, shuffle=True, collate_fn=default_data_collator, batch_size=batch_size, pin_memory=True\n)\neval_dataloader = DataLoader(eval_ds, collate_fn=default_data_collator, batch_size=batch_size, pin_memory=True)\n```\n\n## Model\n\nNow you can load a pretrained model to use as the base model for IA3. This guide uses the [bigscience/mt0-large](https://huggingface.co/bigscience/mt0-large) model, but you can use any sequence-to-sequence model you like.\n\n```py\nfrom transformers import AutoModelForSeq2SeqLM\n\nmodel = AutoModelForSeq2SeqLM.from_pretrained(\"bigscience/mt0-large\")\n```\n\n### PEFT configuration and model\n\nAll PEFT methods need a configuration that contains and specifies all the parameters for how the PEFT method should be applied. Create an [`IA3Config`] with the task type and set the inference mode to `False`. You can find additional parameters for this configuration in the [API reference](../package_reference/ia3#ia3config).\n\n<Tip>\n\nCall the [`~PeftModel.print_trainable_parameters`] method to compare the number of trainable parameters of [`PeftModel`] versus the number of parameters in the base model!\n\n</Tip>\n\nOnce the configuration is setup, pass it to the [`get_peft_model`] function along with the base model to create a trainable [`PeftModel`].\n\n```py\nfrom peft import IA3Config, get_peft_model\n\npeft_config = IA3Config(task_type=\"SEQ_2_SEQ_LM\")\nmodel = get_peft_model(model, peft_config)\nmodel.print_trainable_parameters()\n\"trainable params: 282,624 || all params: 1,229,863,936 || trainable%: 0.022980103060766553\"\n```\n\n### Training\n\nSet up an optimizer and learning rate scheduler.\n\n```py\nimport torch\nfrom transformers import get_linear_schedule_with_warmup\n\nlr = 8e-3\nnum_epochs = 3\n\noptimizer = torch.optim.AdamW(model.parameters(), lr=lr)\nlr_scheduler = get_linear_schedule_with_warmup(\n    optimizer=optimizer,\n    num_warmup_steps=0,\n    num_training_steps=(len(train_dataloader) * num_epochs),\n)\n```\n\nMove the model to the GPU and create a training loop that reports the loss and perplexity for each epoch."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: 126a6414-bfdd-43bb-b083-8184e526d64f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/task_guides/ia3", "title": "IA3", "tokens": 2278, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"126a6414-bfdd-43bb-b083-8184e526d64f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/ia3\", \"title\": \"IA3\", \"tokens\": 2278, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"abaf329e-104c-41c1-9818-3eacccc542f6\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/ia3\", \"title\": \"IA3\", \"tokens\": 2278, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"c1393fd481102f1fef41ff3d4b2f88fbdf1a56a876eaf96f6918e3274864df49\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f01c968e-3d7a-42a2-b701-fcfab7f2b713\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/ia3\", \"title\": \"IA3\", \"tokens\": 2278, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"82338673aea5b0201b7657143ac9a94d4fb6193bfdd6661ef198a907d703a479\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ebd14ddb-a67b-40c8-8c7a-296162977df0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"09fffd659c933e540d8afdcb8f7b7cbb01455de1887e42818ce8864db1a3e12c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4649, \"end_char_idx\": 7710, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "abaf329e-104c-41c1-9818-3eacccc542f6", "doc_id": "abaf329e-104c-41c1-9818-3eacccc542f6", "ref_doc_id": "abaf329e-104c-41c1-9818-3eacccc542f6", "chroma:document": "```py\nfrom transformers import AutoModelForSeq2SeqLM\n\nmodel = AutoModelForSeq2SeqLM.from_pretrained(\"bigscience/mt0-large\")\n```\n\n### PEFT configuration and model\n\nAll PEFT methods need a configuration that contains and specifies all the parameters for how the PEFT method should be applied. Create an [`IA3Config`] with the task type and set the inference mode to `False`. You can find additional parameters for this configuration in the [API reference](../package_reference/ia3#ia3config).\n\n<Tip>\n\nCall the [`~PeftModel.print_trainable_parameters`] method to compare the number of trainable parameters of [`PeftModel`] versus the number of parameters in the base model!\n\n</Tip>\n\nOnce the configuration is setup, pass it to the [`get_peft_model`] function along with the base model to create a trainable [`PeftModel`].\n\n```py\nfrom peft import IA3Config, get_peft_model\n\npeft_config = IA3Config(task_type=\"SEQ_2_SEQ_LM\")\nmodel = get_peft_model(model, peft_config)\nmodel.print_trainable_parameters()\n\"trainable params: 282,624 || all params: 1,229,863,936 || trainable%: 0.022980103060766553\"\n```\n\n### Training\n\nSet up an optimizer and learning rate scheduler.\n\n```py\nimport torch\nfrom transformers import get_linear_schedule_with_warmup\n\nlr = 8e-3\nnum_epochs = 3\n\noptimizer = torch.optim.AdamW(model.parameters(), lr=lr)\nlr_scheduler = get_linear_schedule_with_warmup(\n    optimizer=optimizer,\n    num_warmup_steps=0,\n    num_training_steps=(len(train_dataloader) * num_epochs),\n)\n```\n\nMove the model to the GPU and create a training loop that reports the loss and perplexity for each epoch.\n\n```py\nfrom tqdm import tqdm\n\ndevice = \"cuda\"\nmodel = model.to(device)\n\nfor epoch in range(num_epochs):\n    model.train()\n    total_loss = 0\n    for step, batch in enumerate(tqdm(train_dataloader)):\n        batch = {k: v.to(device) for k, v in batch.items()}\n        outputs = model(**batch)\n        loss = outputs.loss\n        total_loss += loss.detach().float()\n        loss.backward()\n        optimizer.step()\n        lr_scheduler.step()\n        optimizer.zero_grad()\n\n    model.eval()\n    eval_loss = 0\n    eval_preds = []\n    for step, batch in enumerate(tqdm(eval_dataloader)):\n        batch = {k: v.to(device) for k, v in batch.items()}\n        with torch.no_grad():\n            outputs = model(**batch)\n        loss = outputs.loss\n        eval_loss += loss.detach().float()\n        eval_preds.extend(\n            tokenizer.batch_decode(torch.argmax(outputs.logits, -1).detach().cpu().numpy(), skip_special_tokens=True)\n        )\n\n    eval_epoch_loss = eval_loss / len(eval_dataloader)\n    eval_ppl = torch.exp(eval_epoch_loss)\n    train_epoch_loss = total_loss / len(train_dataloader)\n    train_ppl = torch.exp(train_epoch_loss)\n    print(f\"{epoch=}: {train_ppl=} {train_epoch_loss=} {eval_ppl=} {eval_epoch_loss=}\")\n```\n\n## Share your model\n\nAfter training is complete, you can upload your model to the Hub with the [`~transformers.PreTrainedModel.push_to_hub`] method. You'll need to login to your Hugging Face account first and enter your token when prompted."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: ebd14ddb-a67b-40c8-8c7a-296162977df0
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/task_guides/ia3", "title": "IA3", "tokens": 2278, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"ebd14ddb-a67b-40c8-8c7a-296162977df0\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/ia3\", \"title\": \"IA3\", \"tokens\": 2278, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"abaf329e-104c-41c1-9818-3eacccc542f6\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/ia3\", \"title\": \"IA3\", \"tokens\": 2278, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"c1393fd481102f1fef41ff3d4b2f88fbdf1a56a876eaf96f6918e3274864df49\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"126a6414-bfdd-43bb-b083-8184e526d64f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/ia3\", \"title\": \"IA3\", \"tokens\": 2278, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"67f4dbd4db014cc3d97d25e20e220f74d52b2a809e9c5694499f2bc226293c21\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6241, \"end_char_idx\": 8907, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "abaf329e-104c-41c1-9818-3eacccc542f6", "doc_id": "abaf329e-104c-41c1-9818-3eacccc542f6", "ref_doc_id": "abaf329e-104c-41c1-9818-3eacccc542f6", "chroma:document": "```py\nfrom tqdm import tqdm\n\ndevice = \"cuda\"\nmodel = model.to(device)\n\nfor epoch in range(num_epochs):\n    model.train()\n    total_loss = 0\n    for step, batch in enumerate(tqdm(train_dataloader)):\n        batch = {k: v.to(device) for k, v in batch.items()}\n        outputs = model(**batch)\n        loss = outputs.loss\n        total_loss += loss.detach().float()\n        loss.backward()\n        optimizer.step()\n        lr_scheduler.step()\n        optimizer.zero_grad()\n\n    model.eval()\n    eval_loss = 0\n    eval_preds = []\n    for step, batch in enumerate(tqdm(eval_dataloader)):\n        batch = {k: v.to(device) for k, v in batch.items()}\n        with torch.no_grad():\n            outputs = model(**batch)\n        loss = outputs.loss\n        eval_loss += loss.detach().float()\n        eval_preds.extend(\n            tokenizer.batch_decode(torch.argmax(outputs.logits, -1).detach().cpu().numpy(), skip_special_tokens=True)\n        )\n\n    eval_epoch_loss = eval_loss / len(eval_dataloader)\n    eval_ppl = torch.exp(eval_epoch_loss)\n    train_epoch_loss = total_loss / len(train_dataloader)\n    train_ppl = torch.exp(train_epoch_loss)\n    print(f\"{epoch=}: {train_ppl=} {train_epoch_loss=} {eval_ppl=} {eval_epoch_loss=}\")\n```\n\n## Share your model\n\nAfter training is complete, you can upload your model to the Hub with the [`~transformers.PreTrainedModel.push_to_hub`] method. You'll need to login to your Hugging Face account first and enter your token when prompted.\n\n```py\nfrom huggingface_hub import notebook_login\n\naccount = <your-hf-account-name>\npeft_model_id = f\"{account}/mt0-large-ia3\"\nmodel.push_to_hub(peft_model_id)\n```\n\n## Inference\n\nTo load the model for inference, use the [`~AutoPeftModelForSeq2SeqLM.from_pretrained`] method. Let's also load a sentence of financial news from the dataset to generate a sentiment for.\n\n```py\nfrom peft import AutoPeftModelForSeq2SeqLM\n\nmodel = AutoPeftModelForSeq2SeqLM.from_pretrained(\"<your-hf-account-name>/mt0-large-ia3\").to(\"cuda\")\ntokenizer = AutoTokenizer.from_pretrained(\"bigscience/mt0-large\")\n\ni = 15\ninputs = tokenizer(ds[\"validation\"][text_column][i], return_tensors=\"pt\")\nprint(ds[\"validation\"][text_column][i])\n\"The robust growth was the result of the inclusion of clothing chain Lindex in the Group in December 2007 .\"\n```\n\nCall the [`~transformers.GenerationMixin.generate`] method to generate the predicted sentiment label.\n\n```py\nwith torch.no_grad():\n    inputs = {k: v.to(device) for k, v in inputs.items()}\n    outputs = model.generate(input_ids=inputs[\"input_ids\"], max_new_tokens=10)\n    print(tokenizer.batch_decode(outputs.detach().cpu().numpy(), skip_special_tokens=True))\n['positive']\n```"}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: 495f4616-c719-4247-b5ed-a8ba99a4f67f
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/task_guides/lora_based_methods", "title": "LoRA methods", "tokens": 3753, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"495f4616-c719-4247-b5ed-a8ba99a4f67f\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/lora_based_methods\", \"title\": \"LoRA methods\", \"tokens\": 3753, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dfed15ce-d64d-4845-a758-545bd8ee2e21\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/lora_based_methods\", \"title\": \"LoRA methods\", \"tokens\": 3753, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"634a45194657e2d9bc796beab5e721c46721a9c47f7b9c84ae7892683ad7df76\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"48235385-3b98-40b1-854b-97cf56b5dbc6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ee9a75e0d47f903f3b306bd38edf76828f7f9abb73c25179587e27fc7f295773\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3269, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dfed15ce-d64d-4845-a758-545bd8ee2e21", "doc_id": "dfed15ce-d64d-4845-a758-545bd8ee2e21", "ref_doc_id": "dfed15ce-d64d-4845-a758-545bd8ee2e21", "chroma:document": "# LoRA methods\n\nA popular way to efficiently train large models is to insert (typically in the attention blocks) smaller trainable matrices that are a low-rank decomposition of the delta weight matrix to be learnt during finetuning. The pretrained model's original weight matrix is frozen and only the smaller matrices are updated during training. This reduces the number of trainable parameters, reducing memory usage and training time which can be very expensive for large models.\n\nThere are several different ways to express the weight matrix as a low-rank decomposition, but [Low-Rank Adaptation (LoRA)](../conceptual_guides/adapter#low-rank-adaptation-lora) is the most common method. The PEFT library supports several other LoRA variants, such as [Low-Rank Hadamard Product (LoHa)](../conceptual_guides/adapter#low-rank-hadamard-product-loha), [Low-Rank Kronecker Product (LoKr)](../conceptual_guides/adapter#low-rank-kronecker-product-lokr), and [Adaptive Low-Rank Adaptation (AdaLoRA)](../conceptual_guides/adapter#adaptive-low-rank-adaptation-adalora). You can learn more about how these methods work conceptually in the [Adapters](../conceptual_guides/adapter) guide. If you're interested in applying these methods to other tasks and use cases like semantic segmentation, token classification, take a look at our [notebook collection](https://huggingface.co/collections/PEFT/notebooks-6573b28b33e5a4bf5b157fc1)!\n\nAdditionally, PEFT supports the [X-LoRA](../conceptual_guides/adapter#mixture-of-lora-experts-x-lora) Mixture of LoRA Experts method.\n\nThis guide will show you how to quickly train an image classification model - with a low-rank decomposition method - to identify the class of food shown in an image.\n\n<Tip>\n\nSome familiarity with the general process of training an image classification model would be really helpful and allow you to focus on the low-rank decomposition methods. If you're new, we recommend taking a look at the [Image classification](https://huggingface.co/docs/transformers/tasks/image_classification) guide first from the Transformers documentation. When you're ready, come back and see how easy it is to drop PEFT in to your training!\n\n</Tip>\n\nBefore you begin, make sure you have all the necessary libraries installed.\n\n```bash\npip install -q peft transformers datasets\n```\n\n## Dataset\n\nIn this guide, you'll use the [Food-101](https://huggingface.co/datasets/food101) dataset which contains images of 101 food classes (take a look at the [dataset viewer](https://huggingface.co/datasets/food101/viewer/default/train) to get a better idea of what the dataset looks like).\n\nLoad the dataset with the [`~datasets.load_dataset`] function.\n\n```py\nfrom datasets import load_dataset\n\nds = load_dataset(\"food101\")\n```\n\nEach food class is labeled with an integer, so to make it easier to understand what these integers represent, you'll create a `label2id` and `id2label` dictionary to map the integer to its class label.\n\n```py\nlabels = ds[\"train\"].features[\"label\"].names\nlabel2id, id2label = dict(), dict()\nfor i, label in enumerate(labels):\n    label2id[label] = i\n    id2label[i] = label\n\nid2label[2]\n\"baklava\"\n```\n\nLoad an image processor to properly resize and normalize the pixel values of the training and evaluation images."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: 48235385-3b98-40b1-854b-97cf56b5dbc6
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/task_guides/lora_based_methods", "title": "LoRA methods", "tokens": 3753, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"48235385-3b98-40b1-854b-97cf56b5dbc6\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/lora_based_methods\", \"title\": \"LoRA methods\", \"tokens\": 3753, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dfed15ce-d64d-4845-a758-545bd8ee2e21\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/lora_based_methods\", \"title\": \"LoRA methods\", \"tokens\": 3753, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"634a45194657e2d9bc796beab5e721c46721a9c47f7b9c84ae7892683ad7df76\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"495f4616-c719-4247-b5ed-a8ba99a4f67f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/lora_based_methods\", \"title\": \"LoRA methods\", \"tokens\": 3753, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"d043b2c3368d4402a9e1b31229f23f91236d1d2d15ab4c64f274232ec9f1424c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"057899b3-4130-460b-819c-1b9b7fe8d192\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6f9b86f39cf38b8a18144ce8f8b4db211e34ff476de99b4c8248ef71ba1faf79\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1558, \"end_char_idx\": 4912, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dfed15ce-d64d-4845-a758-545bd8ee2e21", "doc_id": "dfed15ce-d64d-4845-a758-545bd8ee2e21", "ref_doc_id": "dfed15ce-d64d-4845-a758-545bd8ee2e21", "chroma:document": "This guide will show you how to quickly train an image classification model - with a low-rank decomposition method - to identify the class of food shown in an image.\n\n<Tip>\n\nSome familiarity with the general process of training an image classification model would be really helpful and allow you to focus on the low-rank decomposition methods. If you're new, we recommend taking a look at the [Image classification](https://huggingface.co/docs/transformers/tasks/image_classification) guide first from the Transformers documentation. When you're ready, come back and see how easy it is to drop PEFT in to your training!\n\n</Tip>\n\nBefore you begin, make sure you have all the necessary libraries installed.\n\n```bash\npip install -q peft transformers datasets\n```\n\n## Dataset\n\nIn this guide, you'll use the [Food-101](https://huggingface.co/datasets/food101) dataset which contains images of 101 food classes (take a look at the [dataset viewer](https://huggingface.co/datasets/food101/viewer/default/train) to get a better idea of what the dataset looks like).\n\nLoad the dataset with the [`~datasets.load_dataset`] function.\n\n```py\nfrom datasets import load_dataset\n\nds = load_dataset(\"food101\")\n```\n\nEach food class is labeled with an integer, so to make it easier to understand what these integers represent, you'll create a `label2id` and `id2label` dictionary to map the integer to its class label.\n\n```py\nlabels = ds[\"train\"].features[\"label\"].names\nlabel2id, id2label = dict(), dict()\nfor i, label in enumerate(labels):\n    label2id[label] = i\n    id2label[i] = label\n\nid2label[2]\n\"baklava\"\n```\n\nLoad an image processor to properly resize and normalize the pixel values of the training and evaluation images.\n\n```py\nfrom transformers import AutoImageProcessor\n\nimage_processor = AutoImageProcessor.from_pretrained(\"google/vit-base-patch16-224-in21k\")\n```\n\nYou can also use the image processor to prepare some transformation functions for data augmentation and pixel scaling.\n\n```py\nfrom torchvision.transforms import (\n    CenterCrop,\n    Compose,\n    Normalize,\n    RandomHorizontalFlip,\n    RandomResizedCrop,\n    Resize,\n    ToTensor,\n)\n\nnormalize = Normalize(mean=image_processor.image_mean, std=image_processor.image_std)\ntrain_transforms = Compose(\n    [\n        RandomResizedCrop(image_processor.size[\"height\"]),\n        RandomHorizontalFlip(),\n        ToTensor(),\n        normalize,\n    ]\n)\n\nval_transforms = Compose(\n    [\n        Resize(image_processor.size[\"height\"]),\n        CenterCrop(image_processor.size[\"height\"]),\n        ToTensor(),\n        normalize,\n    ]\n)\n\ndef preprocess_train(example_batch):\n    example_batch[\"pixel_values\"] = [train_transforms(image.convert(\"RGB\")) for image in example_batch[\"image\"]]\n    return example_batch\n\ndef preprocess_val(example_batch):\n    example_batch[\"pixel_values\"] = [val_transforms(image.convert(\"RGB\")) for image in example_batch[\"image\"]]\n    return example_batch\n```\n\nDefine the training and validation datasets, and use the [`~datasets.Dataset.set_transform`] function to apply the transformations on-the-fly.\n\n```py\ntrain_ds = ds[\"train\"]\nval_ds = ds[\"validation\"]\n\ntrain_ds.set_transform(preprocess_train)\nval_ds.set_transform(preprocess_val)\n```\n\nFinally, you'll need a data collator to create a batch of training and evaluation data and convert the labels to `torch.tensor` objects."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: 057899b3-4130-460b-819c-1b9b7fe8d192
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/task_guides/lora_based_methods", "title": "LoRA methods", "tokens": 3753, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"057899b3-4130-460b-819c-1b9b7fe8d192\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/lora_based_methods\", \"title\": \"LoRA methods\", \"tokens\": 3753, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dfed15ce-d64d-4845-a758-545bd8ee2e21\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/lora_based_methods\", \"title\": \"LoRA methods\", \"tokens\": 3753, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"634a45194657e2d9bc796beab5e721c46721a9c47f7b9c84ae7892683ad7df76\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"48235385-3b98-40b1-854b-97cf56b5dbc6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/lora_based_methods\", \"title\": \"LoRA methods\", \"tokens\": 3753, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"bed5d2681922f43ab00e2d59f74cfe4c0622f399624eef00451aa6277008c221\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1ff2d5b7-7f3c-4d1e-94c0-001da5add902\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5234f433ba166ba12c7d90e5d524f2d18d337218391f23c6552e8f33b14e4181\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3271, \"end_char_idx\": 6693, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dfed15ce-d64d-4845-a758-545bd8ee2e21", "doc_id": "dfed15ce-d64d-4845-a758-545bd8ee2e21", "ref_doc_id": "dfed15ce-d64d-4845-a758-545bd8ee2e21", "chroma:document": "```py\nfrom transformers import AutoImageProcessor\n\nimage_processor = AutoImageProcessor.from_pretrained(\"google/vit-base-patch16-224-in21k\")\n```\n\nYou can also use the image processor to prepare some transformation functions for data augmentation and pixel scaling.\n\n```py\nfrom torchvision.transforms import (\n    CenterCrop,\n    Compose,\n    Normalize,\n    RandomHorizontalFlip,\n    RandomResizedCrop,\n    Resize,\n    ToTensor,\n)\n\nnormalize = Normalize(mean=image_processor.image_mean, std=image_processor.image_std)\ntrain_transforms = Compose(\n    [\n        RandomResizedCrop(image_processor.size[\"height\"]),\n        RandomHorizontalFlip(),\n        ToTensor(),\n        normalize,\n    ]\n)\n\nval_transforms = Compose(\n    [\n        Resize(image_processor.size[\"height\"]),\n        CenterCrop(image_processor.size[\"height\"]),\n        ToTensor(),\n        normalize,\n    ]\n)\n\ndef preprocess_train(example_batch):\n    example_batch[\"pixel_values\"] = [train_transforms(image.convert(\"RGB\")) for image in example_batch[\"image\"]]\n    return example_batch\n\ndef preprocess_val(example_batch):\n    example_batch[\"pixel_values\"] = [val_transforms(image.convert(\"RGB\")) for image in example_batch[\"image\"]]\n    return example_batch\n```\n\nDefine the training and validation datasets, and use the [`~datasets.Dataset.set_transform`] function to apply the transformations on-the-fly.\n\n```py\ntrain_ds = ds[\"train\"]\nval_ds = ds[\"validation\"]\n\ntrain_ds.set_transform(preprocess_train)\nval_ds.set_transform(preprocess_val)\n```\n\nFinally, you'll need a data collator to create a batch of training and evaluation data and convert the labels to `torch.tensor` objects.\n\n```py\nimport torch\n\ndef collate_fn(examples):\n    pixel_values = torch.stack([example[\"pixel_values\"] for example in examples])\n    labels = torch.tensor([example[\"label\"] for example in examples])\n    return {\"pixel_values\": pixel_values, \"labels\": labels}\n```\n\n## Model\n\nNow let's load a pretrained model to use as the base model. This guide uses the [google/vit-base-patch16-224-in21k](https://huggingface.co/google/vit-base-patch16-224-in21k) model, but you can use any image classification model you want. Pass the `label2id` and `id2label` dictionaries to the model so it knows how to map the integer labels to their class labels, and you can optionally pass the `ignore_mismatched_sizes=True` parameter if you're finetuning a checkpoint that has already been finetuned.\n\n```py\nfrom transformers import AutoModelForImageClassification, TrainingArguments, Trainer\n\nmodel = AutoModelForImageClassification.from_pretrained(\n    \"google/vit-base-patch16-224-in21k\",\n    label2id=label2id,\n    id2label=id2label,\n    ignore_mismatched_sizes=True,\n)\n```\n\n### PEFT configuration and model\n\nEvery PEFT method requires a configuration that holds all the parameters specifying how the PEFT method should be applied. Once the configuration is setup, pass it to the [`~peft.get_peft_model`] function along with the base model to create a trainable [`PeftModel`].\n\n<Tip>\n\nCall the [`~PeftModel.print_trainable_parameters`] method to compare the number of parameters of [`PeftModel`] versus the number of parameters in the base model!\n\n</Tip>\n\n<hfoptions id=\"loras\">\n<hfoption id=\"LoRA\">\n\n[LoRA](../conceptual_guides/adapter#low-rank-adaptation-lora) decomposes the weight update matrix into *two* smaller matrices. The size of these low-rank matrices is determined by its *rank* or `r`."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: 1ff2d5b7-7f3c-4d1e-94c0-001da5add902
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/task_guides/lora_based_methods", "title": "LoRA methods", "tokens": 3753, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"1ff2d5b7-7f3c-4d1e-94c0-001da5add902\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/lora_based_methods\", \"title\": \"LoRA methods\", \"tokens\": 3753, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dfed15ce-d64d-4845-a758-545bd8ee2e21\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/lora_based_methods\", \"title\": \"LoRA methods\", \"tokens\": 3753, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"634a45194657e2d9bc796beab5e721c46721a9c47f7b9c84ae7892683ad7df76\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"057899b3-4130-460b-819c-1b9b7fe8d192\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/lora_based_methods\", \"title\": \"LoRA methods\", \"tokens\": 3753, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"7a904ee3df39ba2f261c95d45f32a9adf31661658f172c0f59ae104db5a57cbe\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f57f2776-f517-4a68-85f1-67f8e4af5a0b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"45aaf23d759cb88f411a683547317bb4087060739fd1c7b695f7e89aaa445000\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5247, \"end_char_idx\": 8294, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dfed15ce-d64d-4845-a758-545bd8ee2e21", "doc_id": "dfed15ce-d64d-4845-a758-545bd8ee2e21", "ref_doc_id": "dfed15ce-d64d-4845-a758-545bd8ee2e21", "chroma:document": "This guide uses the [google/vit-base-patch16-224-in21k](https://huggingface.co/google/vit-base-patch16-224-in21k) model, but you can use any image classification model you want. Pass the `label2id` and `id2label` dictionaries to the model so it knows how to map the integer labels to their class labels, and you can optionally pass the `ignore_mismatched_sizes=True` parameter if you're finetuning a checkpoint that has already been finetuned.\n\n```py\nfrom transformers import AutoModelForImageClassification, TrainingArguments, Trainer\n\nmodel = AutoModelForImageClassification.from_pretrained(\n    \"google/vit-base-patch16-224-in21k\",\n    label2id=label2id,\n    id2label=id2label,\n    ignore_mismatched_sizes=True,\n)\n```\n\n### PEFT configuration and model\n\nEvery PEFT method requires a configuration that holds all the parameters specifying how the PEFT method should be applied. Once the configuration is setup, pass it to the [`~peft.get_peft_model`] function along with the base model to create a trainable [`PeftModel`].\n\n<Tip>\n\nCall the [`~PeftModel.print_trainable_parameters`] method to compare the number of parameters of [`PeftModel`] versus the number of parameters in the base model!\n\n</Tip>\n\n<hfoptions id=\"loras\">\n<hfoption id=\"LoRA\">\n\n[LoRA](../conceptual_guides/adapter#low-rank-adaptation-lora) decomposes the weight update matrix into *two* smaller matrices. The size of these low-rank matrices is determined by its *rank* or `r`. A higher rank means the model has more parameters to train, but it also means the model has more learning capacity. You'll also want to specify the `target_modules` which determine where the smaller matrices are inserted. For this guide, you'll target the *query* and *value* matrices of the attention blocks. Other important parameters to set are `lora_alpha` (scaling factor), `bias` (whether `none`, `all` or only the LoRA bias parameters should be trained), and `modules_to_save` (the modules apart from the LoRA layers to be trained and saved). All of these parameters - and more - are found in the [`LoraConfig`].\n\n```py\nfrom peft import LoraConfig, get_peft_model\n\nconfig = LoraConfig(\n    r=16,\n    lora_alpha=16,\n    target_modules=[\"query\", \"value\"],\n    lora_dropout=0.1,\n    bias=\"none\",\n    modules_to_save=[\"classifier\"],\n)\nmodel = get_peft_model(model, config)\nmodel.print_trainable_parameters()\n\"trainable params: 667,493 || all params: 86,543,818 || trainable%: 0.7712775047664294\"\n```\n\n</hfoption>\n<hfoption id=\"LoHa\">\n\n[LoHa](../conceptual_guides/adapter#low-rank-hadamard-product-loha) decomposes the weight update matrix into *four* smaller matrices and each pair of smaller matrices is combined with the Hadamard product. This allows the weight update matrix to keep the same number of trainable parameters when compared to LoRA, but with a higher rank (`r^2` for LoHA when compared to `2*r` for LoRA). The size of the smaller matrices is determined by its *rank* or `r`. You'll also want to specify the `target_modules` which determines where the smaller matrices are inserted."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: f57f2776-f517-4a68-85f1-67f8e4af5a0b
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/task_guides/lora_based_methods", "title": "LoRA methods", "tokens": 3753, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"f57f2776-f517-4a68-85f1-67f8e4af5a0b\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/lora_based_methods\", \"title\": \"LoRA methods\", \"tokens\": 3753, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dfed15ce-d64d-4845-a758-545bd8ee2e21\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/lora_based_methods\", \"title\": \"LoRA methods\", \"tokens\": 3753, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"634a45194657e2d9bc796beab5e721c46721a9c47f7b9c84ae7892683ad7df76\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1ff2d5b7-7f3c-4d1e-94c0-001da5add902\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/lora_based_methods\", \"title\": \"LoRA methods\", \"tokens\": 3753, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"57dff8ec234928044234e051108ecf4d60ed09313087828230bcdd1026d6ed2d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e02cdbe2-3e68-4876-9f08-db9cc65696a4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"dcf4d741c531d0223177fe5e468c8ba699b284d241e578aa7d38203ad84198fa\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6810, \"end_char_idx\": 9727, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dfed15ce-d64d-4845-a758-545bd8ee2e21", "doc_id": "dfed15ce-d64d-4845-a758-545bd8ee2e21", "ref_doc_id": "dfed15ce-d64d-4845-a758-545bd8ee2e21", "chroma:document": "You'll also want to specify the `target_modules` which determine where the smaller matrices are inserted. For this guide, you'll target the *query* and *value* matrices of the attention blocks. Other important parameters to set are `lora_alpha` (scaling factor), `bias` (whether `none`, `all` or only the LoRA bias parameters should be trained), and `modules_to_save` (the modules apart from the LoRA layers to be trained and saved). All of these parameters - and more - are found in the [`LoraConfig`].\n\n```py\nfrom peft import LoraConfig, get_peft_model\n\nconfig = LoraConfig(\n    r=16,\n    lora_alpha=16,\n    target_modules=[\"query\", \"value\"],\n    lora_dropout=0.1,\n    bias=\"none\",\n    modules_to_save=[\"classifier\"],\n)\nmodel = get_peft_model(model, config)\nmodel.print_trainable_parameters()\n\"trainable params: 667,493 || all params: 86,543,818 || trainable%: 0.7712775047664294\"\n```\n\n</hfoption>\n<hfoption id=\"LoHa\">\n\n[LoHa](../conceptual_guides/adapter#low-rank-hadamard-product-loha) decomposes the weight update matrix into *four* smaller matrices and each pair of smaller matrices is combined with the Hadamard product. This allows the weight update matrix to keep the same number of trainable parameters when compared to LoRA, but with a higher rank (`r^2` for LoHA when compared to `2*r` for LoRA). The size of the smaller matrices is determined by its *rank* or `r`. You'll also want to specify the `target_modules` which determines where the smaller matrices are inserted. For this guide, you'll target the *query* and *value* matrices of the attention blocks. Other important parameters to set are `alpha` (scaling factor), and `modules_to_save` (the modules apart from the LoHa layers to be trained and saved). All of these parameters - and more - are found in the [`LoHaConfig`].\n\n```py\nfrom peft import LoHaConfig, get_peft_model\n\nconfig = LoHaConfig(\n    r=16,\n    alpha=16,\n    target_modules=[\"query\", \"value\"],\n    module_dropout=0.1,\n    modules_to_save=[\"classifier\"],\n)\nmodel = get_peft_model(model, config)\nmodel.print_trainable_parameters()\n\"trainable params: 1,257,317 || all params: 87,133,642 || trainable%: 1.4429753779831676\"\n```\n\n</hfoption>\n<hfoption id=\"LoKr\">\n\n[LoKr](../conceptual_guides/adapter#low-rank-kronecker-product-lokr) expresses the weight update matrix as a decomposition of a Kronecker product, creating a block matrix that is able to preserve the rank of the original weight matrix. The size of the smaller matrices are determined by its *rank* or `r`. You'll also want to specify the `target_modules` which determines where the smaller matrices are inserted. For this guide, you'll target the *query* and *value* matrices of the attention blocks. Other important parameters to set are `alpha` (scaling factor), and `modules_to_save` (the modules apart from the LoKr layers to be trained and saved). All of these parameters - and more - are found in the [`LoKrConfig`]."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: e02cdbe2-3e68-4876-9f08-db9cc65696a4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/task_guides/lora_based_methods", "title": "LoRA methods", "tokens": 3753, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"e02cdbe2-3e68-4876-9f08-db9cc65696a4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/lora_based_methods\", \"title\": \"LoRA methods\", \"tokens\": 3753, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dfed15ce-d64d-4845-a758-545bd8ee2e21\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/lora_based_methods\", \"title\": \"LoRA methods\", \"tokens\": 3753, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"634a45194657e2d9bc796beab5e721c46721a9c47f7b9c84ae7892683ad7df76\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f57f2776-f517-4a68-85f1-67f8e4af5a0b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/lora_based_methods\", \"title\": \"LoRA methods\", \"tokens\": 3753, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"a60cec58b210b878a9d46eacd0bf5799a8f639cb0acb095670b4117ee19f92d3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3d73af46-4816-4039-834a-df8309176823\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"11aa23c961b7f08180d662eafbebecfadb216bb2cd1f5a086e2e5389e97b4319\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8188, \"end_char_idx\": 10769, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dfed15ce-d64d-4845-a758-545bd8ee2e21", "doc_id": "dfed15ce-d64d-4845-a758-545bd8ee2e21", "ref_doc_id": "dfed15ce-d64d-4845-a758-545bd8ee2e21", "chroma:document": "You'll also want to specify the `target_modules` which determines where the smaller matrices are inserted. For this guide, you'll target the *query* and *value* matrices of the attention blocks. Other important parameters to set are `alpha` (scaling factor), and `modules_to_save` (the modules apart from the LoHa layers to be trained and saved). All of these parameters - and more - are found in the [`LoHaConfig`].\n\n```py\nfrom peft import LoHaConfig, get_peft_model\n\nconfig = LoHaConfig(\n    r=16,\n    alpha=16,\n    target_modules=[\"query\", \"value\"],\n    module_dropout=0.1,\n    modules_to_save=[\"classifier\"],\n)\nmodel = get_peft_model(model, config)\nmodel.print_trainable_parameters()\n\"trainable params: 1,257,317 || all params: 87,133,642 || trainable%: 1.4429753779831676\"\n```\n\n</hfoption>\n<hfoption id=\"LoKr\">\n\n[LoKr](../conceptual_guides/adapter#low-rank-kronecker-product-lokr) expresses the weight update matrix as a decomposition of a Kronecker product, creating a block matrix that is able to preserve the rank of the original weight matrix. The size of the smaller matrices are determined by its *rank* or `r`. You'll also want to specify the `target_modules` which determines where the smaller matrices are inserted. For this guide, you'll target the *query* and *value* matrices of the attention blocks. Other important parameters to set are `alpha` (scaling factor), and `modules_to_save` (the modules apart from the LoKr layers to be trained and saved). All of these parameters - and more - are found in the [`LoKrConfig`].\n\n```py\nfrom peft import LoKrConfig, get_peft_model\n\nconfig = LoKrConfig(\n    r=16,\n    alpha=16,\n    target_modules=[\"query\", \"value\"],\n    module_dropout=0.1,\n    modules_to_save=[\"classifier\"],\n)\nmodel = get_peft_model(model, config)\nmodel.print_trainable_parameters()\n\"trainable params: 116,069 || all params: 87,172,042 || trainable%: 0.13314934162033282\"\n```\n\n</hfoption>\n<hfoption id=\"AdaLoRA\">\n\n[AdaLoRA](../conceptual_guides/adapter#adaptive-low-rank-adaptation-adalora) efficiently manages the LoRA parameter budget by assigning important weight matrices more parameters and pruning less important ones. In contrast, LoRA evenly distributes parameters across all modules. You can control the average desired *rank* or `r` of the matrices, and which modules to apply AdaLoRA to with `target_modules`. Other important parameters to set are `lora_alpha` (scaling factor), and `modules_to_save` (the modules apart from the AdaLoRA layers to be trained and saved). All of these parameters - and more - are found in the [`AdaLoraConfig`]."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: 3d73af46-4816-4039-834a-df8309176823
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/task_guides/lora_based_methods", "title": "LoRA methods", "tokens": 3753, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"3d73af46-4816-4039-834a-df8309176823\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/lora_based_methods\", \"title\": \"LoRA methods\", \"tokens\": 3753, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dfed15ce-d64d-4845-a758-545bd8ee2e21\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/lora_based_methods\", \"title\": \"LoRA methods\", \"tokens\": 3753, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"634a45194657e2d9bc796beab5e721c46721a9c47f7b9c84ae7892683ad7df76\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e02cdbe2-3e68-4876-9f08-db9cc65696a4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/lora_based_methods\", \"title\": \"LoRA methods\", \"tokens\": 3753, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"a9b13fd18131f00ccf6fe0775c681e48e0a9b85b4448c4ab48e2a7fb9f92a119\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ff84c6eb-fc72-4efa-988b-a0e2b44dff1a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5c19c35172336b92d6cd3a0c5d3083e00c246c43d8f4177949b92820a75d2338\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9241, \"end_char_idx\": 12145, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dfed15ce-d64d-4845-a758-545bd8ee2e21", "doc_id": "dfed15ce-d64d-4845-a758-545bd8ee2e21", "ref_doc_id": "dfed15ce-d64d-4845-a758-545bd8ee2e21", "chroma:document": "The size of the smaller matrices are determined by its *rank* or `r`. You'll also want to specify the `target_modules` which determines where the smaller matrices are inserted. For this guide, you'll target the *query* and *value* matrices of the attention blocks. Other important parameters to set are `alpha` (scaling factor), and `modules_to_save` (the modules apart from the LoKr layers to be trained and saved). All of these parameters - and more - are found in the [`LoKrConfig`].\n\n```py\nfrom peft import LoKrConfig, get_peft_model\n\nconfig = LoKrConfig(\n    r=16,\n    alpha=16,\n    target_modules=[\"query\", \"value\"],\n    module_dropout=0.1,\n    modules_to_save=[\"classifier\"],\n)\nmodel = get_peft_model(model, config)\nmodel.print_trainable_parameters()\n\"trainable params: 116,069 || all params: 87,172,042 || trainable%: 0.13314934162033282\"\n```\n\n</hfoption>\n<hfoption id=\"AdaLoRA\">\n\n[AdaLoRA](../conceptual_guides/adapter#adaptive-low-rank-adaptation-adalora) efficiently manages the LoRA parameter budget by assigning important weight matrices more parameters and pruning less important ones. In contrast, LoRA evenly distributes parameters across all modules. You can control the average desired *rank* or `r` of the matrices, and which modules to apply AdaLoRA to with `target_modules`. Other important parameters to set are `lora_alpha` (scaling factor), and `modules_to_save` (the modules apart from the AdaLoRA layers to be trained and saved). All of these parameters - and more - are found in the [`AdaLoraConfig`].\n\n```py\nfrom peft import AdaLoraConfig, get_peft_model\n\nconfig = AdaLoraConfig(\n    r=8,\n    init_r=12,\n    tinit=200,\n    tfinal=1000,\n    deltaT=10,\n    target_modules=[\"query\", \"value\"],\n    modules_to_save=[\"classifier\"],\n)\nmodel = get_peft_model(model, config)\nmodel.print_trainable_parameters()\n\"trainable params: 520,325 || all params: 87,614,722 || trainable%: 0.5938785036606062\"\n```\n\n</hfoption>\n</hfoptions>\n\n### Training\n\nFor training, let's use the [`~transformers.Trainer`] class from Transformers. The [`Trainer`] contains a PyTorch training loop, and when you're ready, call [`~transformers.Trainer.train`] to start training. To customize the training run, configure the training hyperparameters in the [`~transformers.TrainingArguments`] class. With LoRA-like methods, you can afford to use a higher batch size and learning rate.\n\n> [!WARNING]\n> AdaLoRA has an [`~AdaLoraModel.update_and_allocate`] method that should be called at each training step to update the parameter budget and mask, otherwise the adaptation step is not performed. This requires writing a custom training loop or subclassing the [`~transformers.Trainer`] to incorporate this method. As an example, take a look at this [custom training loop](https://github.com/huggingface/peft/blob/912ad41e96e03652cabf47522cd876076f7a0c4f/examples/conditional_generation/peft_adalora_seq2seq.py#L120)."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: ff84c6eb-fc72-4efa-988b-a0e2b44dff1a
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/task_guides/lora_based_methods", "title": "LoRA methods", "tokens": 3753, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"ff84c6eb-fc72-4efa-988b-a0e2b44dff1a\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/lora_based_methods\", \"title\": \"LoRA methods\", \"tokens\": 3753, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dfed15ce-d64d-4845-a758-545bd8ee2e21\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/lora_based_methods\", \"title\": \"LoRA methods\", \"tokens\": 3753, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"634a45194657e2d9bc796beab5e721c46721a9c47f7b9c84ae7892683ad7df76\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3d73af46-4816-4039-834a-df8309176823\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/lora_based_methods\", \"title\": \"LoRA methods\", \"tokens\": 3753, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"4cb69944b4f22d6fa4f69271911c35d29defecbd71930a314176babb2dcb638b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"55a59ec4-dc19-443e-aa67-04bf7b55d963\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"18d6455c80f8ba3a0c8da875d6e060f26dbbaf8689c1e657aa18120fba066376\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10697, \"end_char_idx\": 13490, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dfed15ce-d64d-4845-a758-545bd8ee2e21", "doc_id": "dfed15ce-d64d-4845-a758-545bd8ee2e21", "ref_doc_id": "dfed15ce-d64d-4845-a758-545bd8ee2e21", "chroma:document": "All of these parameters - and more - are found in the [`AdaLoraConfig`].\n\n```py\nfrom peft import AdaLoraConfig, get_peft_model\n\nconfig = AdaLoraConfig(\n    r=8,\n    init_r=12,\n    tinit=200,\n    tfinal=1000,\n    deltaT=10,\n    target_modules=[\"query\", \"value\"],\n    modules_to_save=[\"classifier\"],\n)\nmodel = get_peft_model(model, config)\nmodel.print_trainable_parameters()\n\"trainable params: 520,325 || all params: 87,614,722 || trainable%: 0.5938785036606062\"\n```\n\n</hfoption>\n</hfoptions>\n\n### Training\n\nFor training, let's use the [`~transformers.Trainer`] class from Transformers. The [`Trainer`] contains a PyTorch training loop, and when you're ready, call [`~transformers.Trainer.train`] to start training. To customize the training run, configure the training hyperparameters in the [`~transformers.TrainingArguments`] class. With LoRA-like methods, you can afford to use a higher batch size and learning rate.\n\n> [!WARNING]\n> AdaLoRA has an [`~AdaLoraModel.update_and_allocate`] method that should be called at each training step to update the parameter budget and mask, otherwise the adaptation step is not performed. This requires writing a custom training loop or subclassing the [`~transformers.Trainer`] to incorporate this method. As an example, take a look at this [custom training loop](https://github.com/huggingface/peft/blob/912ad41e96e03652cabf47522cd876076f7a0c4f/examples/conditional_generation/peft_adalora_seq2seq.py#L120).\n\n```py\nfrom transformers import TrainingArguments, Trainer\n\naccount = \"stevhliu\"\npeft_model_id = f\"{account}/google/vit-base-patch16-224-in21k-lora\"\nbatch_size = 128\n\nargs = TrainingArguments(\n    peft_model_id,\n    remove_unused_columns=False,\n    evaluation_strategy=\"epoch\",\n    save_strategy=\"epoch\",\n    learning_rate=5e-3,\n    per_device_train_batch_size=batch_size,\n    gradient_accumulation_steps=4,\n    per_device_eval_batch_size=batch_size,\n    fp16=True,\n    num_train_epochs=5,\n    logging_steps=10,\n    load_best_model_at_end=True,\n    label_names=[\"labels\"],\n)\n```\n\nBegin training with [`~transformers.Trainer.train`].\n\n```py\ntrainer = Trainer(\n    model,\n    args,\n    train_dataset=train_ds,\n    eval_dataset=val_ds,\n    tokenizer=image_processor,\n    data_collator=collate_fn,\n)\ntrainer.train()\n```\n\n## Share your model\n\nOnce training is complete, you can upload your model to the Hub with the [`~transformers.PreTrainedModel.push_to_hub`] method. You\u2019ll need to login to your Hugging Face account first and enter your token when prompted.\n\n```py\nfrom huggingface_hub import notebook_login\n\nnotebook_login()\n```\n\nCall [`~transformers.PreTrainedModel.push_to_hub`] to save your model to your repositoy.\n\n```py\nmodel.push_to_hub(peft_model_id)\n```\n\n## Inference\n\nLet's load the model from the Hub and test it out on a food image."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: 55a59ec4-dc19-443e-aa67-04bf7b55d963
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/task_guides/lora_based_methods", "title": "LoRA methods", "tokens": 3753, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"55a59ec4-dc19-443e-aa67-04bf7b55d963\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/lora_based_methods\", \"title\": \"LoRA methods\", \"tokens\": 3753, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dfed15ce-d64d-4845-a758-545bd8ee2e21\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/lora_based_methods\", \"title\": \"LoRA methods\", \"tokens\": 3753, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"634a45194657e2d9bc796beab5e721c46721a9c47f7b9c84ae7892683ad7df76\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ff84c6eb-fc72-4efa-988b-a0e2b44dff1a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/lora_based_methods\", \"title\": \"LoRA methods\", \"tokens\": 3753, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"1377f49b2db116fa16d3cd7f869274079cfc1ff13154dcd3b9a56d2b930764bb\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12147, \"end_char_idx\": 14701, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dfed15ce-d64d-4845-a758-545bd8ee2e21", "doc_id": "dfed15ce-d64d-4845-a758-545bd8ee2e21", "ref_doc_id": "dfed15ce-d64d-4845-a758-545bd8ee2e21", "chroma:document": "```py\nfrom transformers import TrainingArguments, Trainer\n\naccount = \"stevhliu\"\npeft_model_id = f\"{account}/google/vit-base-patch16-224-in21k-lora\"\nbatch_size = 128\n\nargs = TrainingArguments(\n    peft_model_id,\n    remove_unused_columns=False,\n    evaluation_strategy=\"epoch\",\n    save_strategy=\"epoch\",\n    learning_rate=5e-3,\n    per_device_train_batch_size=batch_size,\n    gradient_accumulation_steps=4,\n    per_device_eval_batch_size=batch_size,\n    fp16=True,\n    num_train_epochs=5,\n    logging_steps=10,\n    load_best_model_at_end=True,\n    label_names=[\"labels\"],\n)\n```\n\nBegin training with [`~transformers.Trainer.train`].\n\n```py\ntrainer = Trainer(\n    model,\n    args,\n    train_dataset=train_ds,\n    eval_dataset=val_ds,\n    tokenizer=image_processor,\n    data_collator=collate_fn,\n)\ntrainer.train()\n```\n\n## Share your model\n\nOnce training is complete, you can upload your model to the Hub with the [`~transformers.PreTrainedModel.push_to_hub`] method. You\u2019ll need to login to your Hugging Face account first and enter your token when prompted.\n\n```py\nfrom huggingface_hub import notebook_login\n\nnotebook_login()\n```\n\nCall [`~transformers.PreTrainedModel.push_to_hub`] to save your model to your repositoy.\n\n```py\nmodel.push_to_hub(peft_model_id)\n```\n\n## Inference\n\nLet's load the model from the Hub and test it out on a food image.\n\n```py\nfrom peft import PeftConfig, PeftModel\nfrom transformers import AutoImageProcessor\nfrom PIL import Image\nimport requests\n\nconfig = PeftConfig.from_pretrained(\"stevhliu/vit-base-patch16-224-in21k-lora\")\nmodel = AutoModelForImageClassification.from_pretrained(\n    config.base_model_name_or_path,\n    label2id=label2id,\n    id2label=id2label,\n    ignore_mismatched_sizes=True,\n)\nmodel = PeftModel.from_pretrained(model, \"stevhliu/vit-base-patch16-224-in21k-lora\")\n\nurl = \"https://huggingface.co/datasets/sayakpaul/sample-datasets/resolve/main/beignets.jpeg\"\nimage = Image.open(requests.get(url, stream=True).raw)\nimage\n```\n\n<div class=\"flex justify-center\">\n    <img src=\"https://huggingface.co/datasets/sayakpaul/sample-datasets/resolve/main/beignets.jpeg\">\n</div>\n\nConvert the image to RGB and return the underlying PyTorch tensors.\n\n```py\nencoding = image_processor(image.convert(\"RGB\"), return_tensors=\"pt\")\n```\n\nNow run the model and return the predicted class!\n\n```py\nwith torch.no_grad():\n    outputs = model(**encoding)\n    logits = outputs.logits\n\npredicted_class_idx = logits.argmax(-1).item()\nprint(\"Predicted class:\", model.config.id2label[predicted_class_idx])\n\"Predicted class: beignets\"\n```"}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: ab7ebabe-29ae-4062-91cd-3a308e9aa9a4
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/task_guides/prompt_based_methods", "title": "Prompt-based methods", "tokens": 3315, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"ab7ebabe-29ae-4062-91cd-3a308e9aa9a4\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/prompt_based_methods\", \"title\": \"Prompt-based methods\", \"tokens\": 3315, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d9e19246-3764-4463-993f-425191e1e412\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/prompt_based_methods\", \"title\": \"Prompt-based methods\", \"tokens\": 3315, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"a4974d540eeb655582d44b88d415425379b0d06f7d5e927999502cad9d07e2c7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1079951e-59c2-4080-ab65-b7812d8378ff\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4c2cd4a4d76068ebb3dc2055c6672419137f99c5dc1781694f5f14a29091a392\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3269, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d9e19246-3764-4463-993f-425191e1e412", "doc_id": "d9e19246-3764-4463-993f-425191e1e412", "ref_doc_id": "d9e19246-3764-4463-993f-425191e1e412", "chroma:document": "# Prompt-based methods\n\nA prompt can describe a task or provide an example of a task you want the model to learn. Instead of manually creating these prompts, soft prompting methods add learnable parameters to the input embeddings that can be optimized for a specific task while keeping the pretrained model's parameters frozen. This makes it both faster and easier to finetune large language models (LLMs) for new downstream tasks.\n\nThe PEFT library supports several types of prompting methods (p-tuning, prefix tuning, prompt tuning) and you can learn more about how these methods work conceptually in the [Soft prompts](../conceptual_guides/prompting) guide. If you're interested in applying these methods to other tasks and use cases, take a look at our [notebook collection](https://huggingface.co/spaces/PEFT/soft-prompting)!\n\nThis guide will show you how to train a causal language model - with a soft prompting method - to *generate a classification* for whether a tweet is a complaint or not.\n\n<Tip>\n\nSome familiarity with the general process of training a causal language model would be really helpful and allow you to focus on the soft prompting methods. If you're new, we recommend taking a look at the [Causal language modeling](https://huggingface.co/docs/transformers/tasks/language_modeling) guide first from the Transformers documentation. When you're ready, come back and see how easy it is to drop PEFT in to your training!\n\n</Tip>\n\nBefore you begin, make sure you have all the necessary libraries installed.\n\n```bash\npip install -q peft transformers datasets\n```\n\n## Dataset\n\nFor this guide, you'll use the `twitter_complaints` subset of the [RAFT](https://huggingface.co/datasets/ought/raft) dataset. The `twitter_complaints` subset contains tweets labeled as `complaint` and `no complaint` and you can check out the [dataset viewer](https://huggingface.co/datasets/ought/raft/viewer/twitter_complaints) for a better idea of what the data looks like.\n\nUse the [`~datasets.load_dataset`] function to load the dataset and create a new `text_label` column so it is easier to understand what the `Label` values, `1` and `2` mean.\n\n```py\nfrom datasets import load_dataset\n\nds = load_dataset(\"ought/raft\", \"twitter_complaints\")\n\nclasses = [k.replace(\"_\", \" \") for k in ds[\"train\"].features[\"Label\"].names]\nds = ds.map(\n    lambda x: {\"text_label\": [classes[label] for label in x[\"Label\"]]},\n    batched=True,\n    num_proc=1,\n)\nds[\"train\"][0]\n{\"Tweet text\": \"@HMRCcustomers No this is my first job\", \"ID\": 0, \"Label\": 2, \"text_label\": \"no complaint\"}\n```\n\nLoad a tokenizer, define the padding token to use, and determine the maximum length of the tokenized label.\n\n```py\nfrom transformers import AutoTokenizer\n\ntokenizer = AutoTokenizer.from_pretrained(\"bigscience/bloomz-560m\")\nif tokenizer.pad_token_id is None:\n    tokenizer.pad_token_id = tokenizer.eos_token_id\ntarget_max_length = max([len(tokenizer(class_label)[\"input_ids\"]) for class_label in classes])\nprint(target_max_length)\n```\n\nCreate a preprocessing function that tokenizes the tweet text and labels, pad the inputs and labels in each batch, create an attention mask, and truncate sequences to the `max_length`. Then convert the `input_ids`, `attention_mask`, and `labels` to PyTorch tensors."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: 1079951e-59c2-4080-ab65-b7812d8378ff
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/task_guides/prompt_based_methods", "title": "Prompt-based methods", "tokens": 3315, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"1079951e-59c2-4080-ab65-b7812d8378ff\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/prompt_based_methods\", \"title\": \"Prompt-based methods\", \"tokens\": 3315, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d9e19246-3764-4463-993f-425191e1e412\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/prompt_based_methods\", \"title\": \"Prompt-based methods\", \"tokens\": 3315, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"a4974d540eeb655582d44b88d415425379b0d06f7d5e927999502cad9d07e2c7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ab7ebabe-29ae-4062-91cd-3a308e9aa9a4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/prompt_based_methods\", \"title\": \"Prompt-based methods\", \"tokens\": 3315, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"5bc59bca928d8806ff532178cb654dc177791f1f7b201d826db18194381f8e20\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"23c9de7e-4830-4ad5-89c7-0d68335cf1a1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"376692206b04986c6f6243510992e45f6e657a8e896b226dcda1da293398f12f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1721, \"end_char_idx\": 4775, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d9e19246-3764-4463-993f-425191e1e412", "doc_id": "d9e19246-3764-4463-993f-425191e1e412", "ref_doc_id": "d9e19246-3764-4463-993f-425191e1e412", "chroma:document": "The `twitter_complaints` subset contains tweets labeled as `complaint` and `no complaint` and you can check out the [dataset viewer](https://huggingface.co/datasets/ought/raft/viewer/twitter_complaints) for a better idea of what the data looks like.\n\nUse the [`~datasets.load_dataset`] function to load the dataset and create a new `text_label` column so it is easier to understand what the `Label` values, `1` and `2` mean.\n\n```py\nfrom datasets import load_dataset\n\nds = load_dataset(\"ought/raft\", \"twitter_complaints\")\n\nclasses = [k.replace(\"_\", \" \") for k in ds[\"train\"].features[\"Label\"].names]\nds = ds.map(\n    lambda x: {\"text_label\": [classes[label] for label in x[\"Label\"]]},\n    batched=True,\n    num_proc=1,\n)\nds[\"train\"][0]\n{\"Tweet text\": \"@HMRCcustomers No this is my first job\", \"ID\": 0, \"Label\": 2, \"text_label\": \"no complaint\"}\n```\n\nLoad a tokenizer, define the padding token to use, and determine the maximum length of the tokenized label.\n\n```py\nfrom transformers import AutoTokenizer\n\ntokenizer = AutoTokenizer.from_pretrained(\"bigscience/bloomz-560m\")\nif tokenizer.pad_token_id is None:\n    tokenizer.pad_token_id = tokenizer.eos_token_id\ntarget_max_length = max([len(tokenizer(class_label)[\"input_ids\"]) for class_label in classes])\nprint(target_max_length)\n```\n\nCreate a preprocessing function that tokenizes the tweet text and labels, pad the inputs and labels in each batch, create an attention mask, and truncate sequences to the `max_length`. Then convert the `input_ids`, `attention_mask`, and `labels` to PyTorch tensors.\n\n```py\nimport torch\n\nmax_length = 64\n\ndef preprocess_function(examples, text_column=\"Tweet text\", label_column=\"text_label\"):\n    batch_size = len(examples[text_column])\n    inputs = [f\"{text_column} : {x} Label : \" for x in examples[text_column]]\n    targets = [str(x) for x in examples[label_column]]\n    model_inputs = tokenizer(inputs)\n    labels = tokenizer(targets)\n    classes = [k.replace(\"_\", \" \") for k in ds[\"train\"].features[\"Label\"].names]\n    for i in range(batch_size):\n        sample_input_ids = model_inputs[\"input_ids\"][i]\n        label_input_ids = labels[\"input_ids\"][i]\n        model_inputs[\"input_ids\"][i] = [tokenizer.pad_token_id] * (\n            max_length - len(sample_input_ids)\n        ) + sample_input_ids\n        model_inputs[\"attention_mask\"][i] = [0] * (max_length - len(sample_input_ids)) + model_inputs[\n            \"attention_mask\"\n        ][i]\n        labels[\"input_ids\"][i] = [-100] * (max_length - len(label_input_ids)) + label_input_ids\n        model_inputs[\"input_ids\"][i] = torch.tensor(model_inputs[\"input_ids\"][i][:max_length])\n        model_inputs[\"attention_mask\"][i] = torch.tensor(model_inputs[\"attention_mask\"][i][:max_length])\n        labels[\"input_ids\"][i] = torch.tensor(labels[\"input_ids\"][i][:max_length])\n    model_inputs[\"labels\"] = labels[\"input_ids\"]\n    return model_inputs\n```\n\nApply the preprocessing function to the entire dataset with the [`~datasets.Dataset.map`] function, and remove the unprocessed columns because the model won't need them."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: 23c9de7e-4830-4ad5-89c7-0d68335cf1a1
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/task_guides/prompt_based_methods", "title": "Prompt-based methods", "tokens": 3315, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"23c9de7e-4830-4ad5-89c7-0d68335cf1a1\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/prompt_based_methods\", \"title\": \"Prompt-based methods\", \"tokens\": 3315, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d9e19246-3764-4463-993f-425191e1e412\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/prompt_based_methods\", \"title\": \"Prompt-based methods\", \"tokens\": 3315, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"a4974d540eeb655582d44b88d415425379b0d06f7d5e927999502cad9d07e2c7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1079951e-59c2-4080-ab65-b7812d8378ff\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/prompt_based_methods\", \"title\": \"Prompt-based methods\", \"tokens\": 3315, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"4e1b04cab24830e861bfc901605792388eb090043b6d5ead117b24c3c36af659\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f6d2ebcf-235d-4a65-b0f0-d7082267550c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a8beb1fae6e98477183ff2817bbdb4efbacc3c28fae9e5bd6c18a4784d856b71\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3189, \"end_char_idx\": 6374, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d9e19246-3764-4463-993f-425191e1e412", "doc_id": "d9e19246-3764-4463-993f-425191e1e412", "ref_doc_id": "d9e19246-3764-4463-993f-425191e1e412", "chroma:document": "Then convert the `input_ids`, `attention_mask`, and `labels` to PyTorch tensors.\n\n```py\nimport torch\n\nmax_length = 64\n\ndef preprocess_function(examples, text_column=\"Tweet text\", label_column=\"text_label\"):\n    batch_size = len(examples[text_column])\n    inputs = [f\"{text_column} : {x} Label : \" for x in examples[text_column]]\n    targets = [str(x) for x in examples[label_column]]\n    model_inputs = tokenizer(inputs)\n    labels = tokenizer(targets)\n    classes = [k.replace(\"_\", \" \") for k in ds[\"train\"].features[\"Label\"].names]\n    for i in range(batch_size):\n        sample_input_ids = model_inputs[\"input_ids\"][i]\n        label_input_ids = labels[\"input_ids\"][i]\n        model_inputs[\"input_ids\"][i] = [tokenizer.pad_token_id] * (\n            max_length - len(sample_input_ids)\n        ) + sample_input_ids\n        model_inputs[\"attention_mask\"][i] = [0] * (max_length - len(sample_input_ids)) + model_inputs[\n            \"attention_mask\"\n        ][i]\n        labels[\"input_ids\"][i] = [-100] * (max_length - len(label_input_ids)) + label_input_ids\n        model_inputs[\"input_ids\"][i] = torch.tensor(model_inputs[\"input_ids\"][i][:max_length])\n        model_inputs[\"attention_mask\"][i] = torch.tensor(model_inputs[\"attention_mask\"][i][:max_length])\n        labels[\"input_ids\"][i] = torch.tensor(labels[\"input_ids\"][i][:max_length])\n    model_inputs[\"labels\"] = labels[\"input_ids\"]\n    return model_inputs\n```\n\nApply the preprocessing function to the entire dataset with the [`~datasets.Dataset.map`] function, and remove the unprocessed columns because the model won't need them.\n\n```py\nprocessed_ds = ds.map(\n    preprocess_function,\n    batched=True,\n    num_proc=1,\n    remove_columns=ds[\"train\"].column_names,\n    load_from_cache_file=False,\n    desc=\"Running tokenizer on dataset\",\n)\n```\n\nFinally, create a training and evaluation [`DataLoader`](https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader). You can set `pin_memory=True` to speed up the data transfer to the GPU during training if the samples in your dataset are on a CPU.\n\n```py\nfrom torch.utils.data import DataLoader\nfrom transformers import default_data_collator\n\ntrain_ds = processed_ds[\"train\"]\neval_ds = processed_ds[\"test\"]\n\nbatch_size = 16\n\ntrain_dataloader = DataLoader(train_ds, shuffle=True, collate_fn=default_data_collator, batch_size=batch_size, pin_memory=True)\neval_dataloader = DataLoader(eval_ds, collate_fn=default_data_collator, batch_size=batch_size, pin_memory=True)\n```\n\n## Model\n\nNow let's load a pretrained model to use as the base model for the soft prompt method. This guide uses the [bigscience/bloomz-560m](https://huggingface.co/bigscience/bloomz-560m) model, but you can use any causal language model you want.\n\n```py\nfrom transformers import AutoModelForCausalLM\n\nmodel = AutoModelForCausalLM.from_pretrained(\"bigscience/bloomz-560m\")\n```\n\n### PEFT configuration and model\n\nFor any PEFT method, you'll need to create a configuration which contains all the parameters that specify how the PEFT method should be applied. Once the configuration is setup, pass it to the [`~peft.get_peft_model`] function along with the base model to create a trainable [`PeftModel`]."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: f6d2ebcf-235d-4a65-b0f0-d7082267550c
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/task_guides/prompt_based_methods", "title": "Prompt-based methods", "tokens": 3315, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"f6d2ebcf-235d-4a65-b0f0-d7082267550c\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/prompt_based_methods\", \"title\": \"Prompt-based methods\", \"tokens\": 3315, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d9e19246-3764-4463-993f-425191e1e412\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/prompt_based_methods\", \"title\": \"Prompt-based methods\", \"tokens\": 3315, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"a4974d540eeb655582d44b88d415425379b0d06f7d5e927999502cad9d07e2c7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"23c9de7e-4830-4ad5-89c7-0d68335cf1a1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/prompt_based_methods\", \"title\": \"Prompt-based methods\", \"tokens\": 3315, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"9626c92ef0347e94fd5e84a1ae191c30a696579d34154994314d257e0a331094\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"617724e7-83b9-48ef-a4cb-04cc865426ab\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"98a8952586c8e21b77efe5436ccb0779dfa9dd309b8d05f1d6240454f9ed85a9\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4777, \"end_char_idx\": 7603, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d9e19246-3764-4463-993f-425191e1e412", "doc_id": "d9e19246-3764-4463-993f-425191e1e412", "ref_doc_id": "d9e19246-3764-4463-993f-425191e1e412", "chroma:document": "```py\nprocessed_ds = ds.map(\n    preprocess_function,\n    batched=True,\n    num_proc=1,\n    remove_columns=ds[\"train\"].column_names,\n    load_from_cache_file=False,\n    desc=\"Running tokenizer on dataset\",\n)\n```\n\nFinally, create a training and evaluation [`DataLoader`](https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader). You can set `pin_memory=True` to speed up the data transfer to the GPU during training if the samples in your dataset are on a CPU.\n\n```py\nfrom torch.utils.data import DataLoader\nfrom transformers import default_data_collator\n\ntrain_ds = processed_ds[\"train\"]\neval_ds = processed_ds[\"test\"]\n\nbatch_size = 16\n\ntrain_dataloader = DataLoader(train_ds, shuffle=True, collate_fn=default_data_collator, batch_size=batch_size, pin_memory=True)\neval_dataloader = DataLoader(eval_ds, collate_fn=default_data_collator, batch_size=batch_size, pin_memory=True)\n```\n\n## Model\n\nNow let's load a pretrained model to use as the base model for the soft prompt method. This guide uses the [bigscience/bloomz-560m](https://huggingface.co/bigscience/bloomz-560m) model, but you can use any causal language model you want.\n\n```py\nfrom transformers import AutoModelForCausalLM\n\nmodel = AutoModelForCausalLM.from_pretrained(\"bigscience/bloomz-560m\")\n```\n\n### PEFT configuration and model\n\nFor any PEFT method, you'll need to create a configuration which contains all the parameters that specify how the PEFT method should be applied. Once the configuration is setup, pass it to the [`~peft.get_peft_model`] function along with the base model to create a trainable [`PeftModel`].\n\n<Tip>\n\nCall the [`~PeftModel.print_trainable_parameters`] method to compare the number of trainable parameters of [`PeftModel`] versus the number of parameters in the base model!\n\n</Tip>\n\n<hfoptions id=\"configurations\">\n<hfoption id=\"p-tuning\">\n\n[P-tuning](../conceptual_guides/prompting#p-tuning) adds a trainable embedding tensor where the prompt tokens can be added anywhere in the input sequence. Create a [`PromptEncoderConfig`] with the task type, the number of virtual tokens to add and learn, and the hidden size of the encoder for learning the prompt parameters.\n\n```py\nfrom peft import PromptEncoderConfig, get_peft_model\n\npeft_config = PromptEncoderConfig(task_type=\"CAUSAL_LM\", num_virtual_tokens=20, encoder_hidden_size=128)\nmodel = get_peft_model(model, peft_config)\nmodel.print_trainable_parameters()\n\"trainable params: 300,288 || all params: 559,514,880 || trainable%: 0.05366935013417338\"\n```\n\n</hfoption>\n<hfoption id=\"prefix tuning\">\n\n[Prefix tuning](../conceptual_guides/prompting#prefix-tuning) adds task-specific parameters in all of the model layers, which are optimized by a separate feed-forward network. Create a [`PrefixTuningConfig`] with the task type and number of virtual tokens to add and learn."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: 617724e7-83b9-48ef-a4cb-04cc865426ab
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/task_guides/prompt_based_methods", "title": "Prompt-based methods", "tokens": 3315, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"617724e7-83b9-48ef-a4cb-04cc865426ab\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/prompt_based_methods\", \"title\": \"Prompt-based methods\", \"tokens\": 3315, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d9e19246-3764-4463-993f-425191e1e412\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/prompt_based_methods\", \"title\": \"Prompt-based methods\", \"tokens\": 3315, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"a4974d540eeb655582d44b88d415425379b0d06f7d5e927999502cad9d07e2c7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f6d2ebcf-235d-4a65-b0f0-d7082267550c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/prompt_based_methods\", \"title\": \"Prompt-based methods\", \"tokens\": 3315, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"3bb2f64c1a783dd34304abbd4f94e339924dd3da7c995bcf5099a9067030c71d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"06e4a745-05be-42d5-847a-d8f4a0778b38\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"49ac478c1e78e5a26434145b27e88748973b6570563f87873588756f129e80ec\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6230, \"end_char_idx\": 8683, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d9e19246-3764-4463-993f-425191e1e412", "doc_id": "d9e19246-3764-4463-993f-425191e1e412", "ref_doc_id": "d9e19246-3764-4463-993f-425191e1e412", "chroma:document": "Once the configuration is setup, pass it to the [`~peft.get_peft_model`] function along with the base model to create a trainable [`PeftModel`].\n\n<Tip>\n\nCall the [`~PeftModel.print_trainable_parameters`] method to compare the number of trainable parameters of [`PeftModel`] versus the number of parameters in the base model!\n\n</Tip>\n\n<hfoptions id=\"configurations\">\n<hfoption id=\"p-tuning\">\n\n[P-tuning](../conceptual_guides/prompting#p-tuning) adds a trainable embedding tensor where the prompt tokens can be added anywhere in the input sequence. Create a [`PromptEncoderConfig`] with the task type, the number of virtual tokens to add and learn, and the hidden size of the encoder for learning the prompt parameters.\n\n```py\nfrom peft import PromptEncoderConfig, get_peft_model\n\npeft_config = PromptEncoderConfig(task_type=\"CAUSAL_LM\", num_virtual_tokens=20, encoder_hidden_size=128)\nmodel = get_peft_model(model, peft_config)\nmodel.print_trainable_parameters()\n\"trainable params: 300,288 || all params: 559,514,880 || trainable%: 0.05366935013417338\"\n```\n\n</hfoption>\n<hfoption id=\"prefix tuning\">\n\n[Prefix tuning](../conceptual_guides/prompting#prefix-tuning) adds task-specific parameters in all of the model layers, which are optimized by a separate feed-forward network. Create a [`PrefixTuningConfig`] with the task type and number of virtual tokens to add and learn.\n\n```py\nfrom peft import PrefixTuningConfig, get_peft_model\n\npeft_config = PrefixTuningConfig(task_type=\"CAUSAL_LM\", num_virtual_tokens=20)\nmodel = get_peft_model(model, peft_config)\nmodel.print_trainable_parameters()\n\"trainable params: 983,040 || all params: 560,197,632 || trainable%: 0.1754809274167014\"\n```\n\n</hfoption>\n<hfoption id=\"prompt tuning\">\n\n[Prompt tuning](../conceptual_guides/prompting#prompt-tuning) formulates all tasks as a *generation* task and it adds a task-specific prompt to the input which is updated independently. The `prompt_tuning_init_text` parameter specifies how to finetune the model (in this case, it is classifying whether tweets are complaints or not). For the best results, the `prompt_tuning_init_text` should have the same number of tokens that should be predicted. To do this, you can set `num_virtual_tokens` to the number of tokens of the `prompt_tuning_init_text`.\n\nCreate a [`PromptTuningConfig`] with the task type, the initial prompt tuning text to train the model with, the number of virtual tokens to add and learn, and a tokenizer."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: 06e4a745-05be-42d5-847a-d8f4a0778b38
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/task_guides/prompt_based_methods", "title": "Prompt-based methods", "tokens": 3315, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"06e4a745-05be-42d5-847a-d8f4a0778b38\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/prompt_based_methods\", \"title\": \"Prompt-based methods\", \"tokens\": 3315, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d9e19246-3764-4463-993f-425191e1e412\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/prompt_based_methods\", \"title\": \"Prompt-based methods\", \"tokens\": 3315, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"a4974d540eeb655582d44b88d415425379b0d06f7d5e927999502cad9d07e2c7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"617724e7-83b9-48ef-a4cb-04cc865426ab\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/prompt_based_methods\", \"title\": \"Prompt-based methods\", \"tokens\": 3315, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"ea38bbcd39c04d2a3b60f3a6914945e159aee7d9ef36c8f95d31bc0bd164a339\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"669e4a27-deee-4bd7-bafc-5357c0cdfdab\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ea34d3af10f0bc11557d802eeb7e06d5dc7e11924cd290f2cad236c948dcc913\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7506, \"end_char_idx\": 9819, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d9e19246-3764-4463-993f-425191e1e412", "doc_id": "d9e19246-3764-4463-993f-425191e1e412", "ref_doc_id": "d9e19246-3764-4463-993f-425191e1e412", "chroma:document": "Create a [`PrefixTuningConfig`] with the task type and number of virtual tokens to add and learn.\n\n```py\nfrom peft import PrefixTuningConfig, get_peft_model\n\npeft_config = PrefixTuningConfig(task_type=\"CAUSAL_LM\", num_virtual_tokens=20)\nmodel = get_peft_model(model, peft_config)\nmodel.print_trainable_parameters()\n\"trainable params: 983,040 || all params: 560,197,632 || trainable%: 0.1754809274167014\"\n```\n\n</hfoption>\n<hfoption id=\"prompt tuning\">\n\n[Prompt tuning](../conceptual_guides/prompting#prompt-tuning) formulates all tasks as a *generation* task and it adds a task-specific prompt to the input which is updated independently. The `prompt_tuning_init_text` parameter specifies how to finetune the model (in this case, it is classifying whether tweets are complaints or not). For the best results, the `prompt_tuning_init_text` should have the same number of tokens that should be predicted. To do this, you can set `num_virtual_tokens` to the number of tokens of the `prompt_tuning_init_text`.\n\nCreate a [`PromptTuningConfig`] with the task type, the initial prompt tuning text to train the model with, the number of virtual tokens to add and learn, and a tokenizer.\n\n```py\nfrom peft import PromptTuningConfig, PromptTuningInit, get_peft_model\n\nprompt_tuning_init_text = \"Classify if the tweet is a complaint or no complaint.\\n\"\npeft_config = PromptTuningConfig(\n    task_type=\"CAUSAL_LM\",\n    prompt_tuning_init=PromptTuningInit.TEXT,\n    num_virtual_tokens=len(tokenizer(prompt_tuning_init_text)[\"input_ids\"]),\n    prompt_tuning_init_text=prompt_tuning_init_text,\n    tokenizer_name_or_path=\"bigscience/bloomz-560m\",\n)\nmodel = get_peft_model(model, peft_config)\nmodel.print_trainable_parameters()\n\"trainable params: 8,192 || all params: 559,222,784 || trainable%: 0.0014648902430985358\"\n```\n\n</hfoption>\n</hfoptions>\n\n### Training\n\nSet up an optimizer and learning rate scheduler.\n\n```py\nfrom transformers import get_linear_schedule_with_warmup\n\nlr = 3e-2\nnum_epochs = 50\n\noptimizer = torch.optim.AdamW(model.parameters(), lr=lr)\nlr_scheduler = get_linear_schedule_with_warmup(\n    optimizer=optimizer,\n    num_warmup_steps=0,\n    num_training_steps=(len(train_dataloader) * num_epochs),\n)\n```\n\nMove the model to the GPU and create a training loop that reports the loss and perplexity for each epoch."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: 669e4a27-deee-4bd7-bafc-5357c0cdfdab
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/task_guides/prompt_based_methods", "title": "Prompt-based methods", "tokens": 3315, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"669e4a27-deee-4bd7-bafc-5357c0cdfdab\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/prompt_based_methods\", \"title\": \"Prompt-based methods\", \"tokens\": 3315, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d9e19246-3764-4463-993f-425191e1e412\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/prompt_based_methods\", \"title\": \"Prompt-based methods\", \"tokens\": 3315, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"a4974d540eeb655582d44b88d415425379b0d06f7d5e927999502cad9d07e2c7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"06e4a745-05be-42d5-847a-d8f4a0778b38\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/prompt_based_methods\", \"title\": \"Prompt-based methods\", \"tokens\": 3315, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"7b5bb5b2b66c31f4fdd5c72f0bef3446c522e47d7301333448ec76635d217bff\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a7bec6ec-e4d9-4bb0-9fd6-3b7f0e9fc529\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4b76fd57498d1d04440f80ba6442bcebe3c0188ba37f2d939ff08dc92be03fa8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8292, \"end_char_idx\": 11289, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d9e19246-3764-4463-993f-425191e1e412", "doc_id": "d9e19246-3764-4463-993f-425191e1e412", "ref_doc_id": "d9e19246-3764-4463-993f-425191e1e412", "chroma:document": "For the best results, the `prompt_tuning_init_text` should have the same number of tokens that should be predicted. To do this, you can set `num_virtual_tokens` to the number of tokens of the `prompt_tuning_init_text`.\n\nCreate a [`PromptTuningConfig`] with the task type, the initial prompt tuning text to train the model with, the number of virtual tokens to add and learn, and a tokenizer.\n\n```py\nfrom peft import PromptTuningConfig, PromptTuningInit, get_peft_model\n\nprompt_tuning_init_text = \"Classify if the tweet is a complaint or no complaint.\\n\"\npeft_config = PromptTuningConfig(\n    task_type=\"CAUSAL_LM\",\n    prompt_tuning_init=PromptTuningInit.TEXT,\n    num_virtual_tokens=len(tokenizer(prompt_tuning_init_text)[\"input_ids\"]),\n    prompt_tuning_init_text=prompt_tuning_init_text,\n    tokenizer_name_or_path=\"bigscience/bloomz-560m\",\n)\nmodel = get_peft_model(model, peft_config)\nmodel.print_trainable_parameters()\n\"trainable params: 8,192 || all params: 559,222,784 || trainable%: 0.0014648902430985358\"\n```\n\n</hfoption>\n</hfoptions>\n\n### Training\n\nSet up an optimizer and learning rate scheduler.\n\n```py\nfrom transformers import get_linear_schedule_with_warmup\n\nlr = 3e-2\nnum_epochs = 50\n\noptimizer = torch.optim.AdamW(model.parameters(), lr=lr)\nlr_scheduler = get_linear_schedule_with_warmup(\n    optimizer=optimizer,\n    num_warmup_steps=0,\n    num_training_steps=(len(train_dataloader) * num_epochs),\n)\n```\n\nMove the model to the GPU and create a training loop that reports the loss and perplexity for each epoch.\n\n```py\nfrom tqdm import tqdm\n\ndevice = \"cuda\"\nmodel = model.to(device)\n\nfor epoch in range(num_epochs):\n    model.train()\n    total_loss = 0\n    for step, batch in enumerate(tqdm(train_dataloader)):\n        batch = {k: v.to(device) for k, v in batch.items()}\n        outputs = model(**batch)\n        loss = outputs.loss\n        total_loss += loss.detach().float()\n        loss.backward()\n        optimizer.step()\n        lr_scheduler.step()\n        optimizer.zero_grad()\n\n    model.eval()\n    eval_loss = 0\n    eval_preds = []\n    for step, batch in enumerate(tqdm(eval_dataloader)):\n        batch = {k: v.to(device) for k, v in batch.items()}\n        with torch.no_grad():\n            outputs = model(**batch)\n        loss = outputs.loss\n        eval_loss += loss.detach().float()\n        eval_preds.extend(\n            tokenizer.batch_decode(torch.argmax(outputs.logits, -1).detach().cpu().numpy(), skip_special_tokens=True)\n        )\n\n    eval_epoch_loss = eval_loss / len(eval_dataloader)\n    eval_ppl = torch.exp(eval_epoch_loss)\n    train_epoch_loss = total_loss / len(train_dataloader)\n    train_ppl = torch.exp(train_epoch_loss)\n    print(f\"{epoch=}: {train_ppl=} {train_epoch_loss=} {eval_ppl=} {eval_epoch_loss=}\")\n```\n\n## Share your model\n\nOnce training is complete, you can upload your model to the Hub with the [`~transformers.PreTrainedModel.push_to_hub`] method. You'll need to login to your Hugging Face account first and enter your token when prompted."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: a7bec6ec-e4d9-4bb0-9fd6-3b7f0e9fc529
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/task_guides/prompt_based_methods", "title": "Prompt-based methods", "tokens": 3315, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"a7bec6ec-e4d9-4bb0-9fd6-3b7f0e9fc529\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/prompt_based_methods\", \"title\": \"Prompt-based methods\", \"tokens\": 3315, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d9e19246-3764-4463-993f-425191e1e412\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/prompt_based_methods\", \"title\": \"Prompt-based methods\", \"tokens\": 3315, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"a4974d540eeb655582d44b88d415425379b0d06f7d5e927999502cad9d07e2c7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"669e4a27-deee-4bd7-bafc-5357c0cdfdab\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/prompt_based_methods\", \"title\": \"Prompt-based methods\", \"tokens\": 3315, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"7ea13b08901d3bc5e583118cc69a4b3e89a2861a940ad7c4ad23c9700773289b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7412ea36-31c0-48aa-8eac-4870dfad9bd1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"be68b9bfe457f9a31ad311f79345e48d928e1ba6583b2d43c8704c9e02280990\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9821, \"end_char_idx\": 12970, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d9e19246-3764-4463-993f-425191e1e412", "doc_id": "d9e19246-3764-4463-993f-425191e1e412", "ref_doc_id": "d9e19246-3764-4463-993f-425191e1e412", "chroma:document": "```py\nfrom tqdm import tqdm\n\ndevice = \"cuda\"\nmodel = model.to(device)\n\nfor epoch in range(num_epochs):\n    model.train()\n    total_loss = 0\n    for step, batch in enumerate(tqdm(train_dataloader)):\n        batch = {k: v.to(device) for k, v in batch.items()}\n        outputs = model(**batch)\n        loss = outputs.loss\n        total_loss += loss.detach().float()\n        loss.backward()\n        optimizer.step()\n        lr_scheduler.step()\n        optimizer.zero_grad()\n\n    model.eval()\n    eval_loss = 0\n    eval_preds = []\n    for step, batch in enumerate(tqdm(eval_dataloader)):\n        batch = {k: v.to(device) for k, v in batch.items()}\n        with torch.no_grad():\n            outputs = model(**batch)\n        loss = outputs.loss\n        eval_loss += loss.detach().float()\n        eval_preds.extend(\n            tokenizer.batch_decode(torch.argmax(outputs.logits, -1).detach().cpu().numpy(), skip_special_tokens=True)\n        )\n\n    eval_epoch_loss = eval_loss / len(eval_dataloader)\n    eval_ppl = torch.exp(eval_epoch_loss)\n    train_epoch_loss = total_loss / len(train_dataloader)\n    train_ppl = torch.exp(train_epoch_loss)\n    print(f\"{epoch=}: {train_ppl=} {train_epoch_loss=} {eval_ppl=} {eval_epoch_loss=}\")\n```\n\n## Share your model\n\nOnce training is complete, you can upload your model to the Hub with the [`~transformers.PreTrainedModel.push_to_hub`] method. You'll need to login to your Hugging Face account first and enter your token when prompted.\n\n```py\nfrom huggingface_hub import notebook_login\n\naccount = <your-hf-account-name>\npeft_model_id = f\"{account}/bloomz-560-m-peft-method\"\nmodel.push_to_hub(peft_model_id)\n```\n\nIf you check the model file size in the repository, you\u2019ll see that it is a lot smaller than a full sized model!\n\n<div class=\"flex flex-col justify-center\">\n  <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/peft/PEFT-hub-screenshot.png\"/>\n  <figcaption class=\"text-center\">For example, the adapter weights for a opt-350m model stored on the Hub are only ~6MB compared to the full model size which can be ~700MB.</figcaption>\n</div>\n\n## Inference\n\nLet's load the model for inference and test it out on a tweet!\n\n```py\nfrom peft import AutoPeftModelForCausalLM\n\nmodel = AutoPeftModelForCausalLM.from_pretrained(\"peft_model_id\").to(\"cuda\")\ntokenizer = AutoTokenizer.from_pretrained(\"bigscience/bloomz-560m\")\n\ni = 15\ninputs = tokenizer(f'{text_column} : {ds[\"test\"][i][\"Tweet text\"]} Label : ', return_tensors=\"pt\")\nprint(ds[\"test\"][i][\"Tweet text\"])\n\"@NYTsupport i have complained a dozen times &amp; yet my papers are still thrown FAR from my door. Why is this so hard to resolve?\"\n```\n\nCall the [`~transformers.GenerationMixin.generate`] method to generate the predicted classification label.\n\n```py\nwith torch.no_grad():\n    inputs = {k: v.to(device) for k, v in inputs.items()}\n    outputs = model.generate(input_ids=inputs[\"input_ids\"], max_new_tokens=10)\n    print(tokenizer.batch_decode(outputs.detach().cpu().numpy(), skip_special_tokens=True))\n\"['Tweet text : @NYTsupport i have complained a dozen times &amp; yet my papers are still thrown FAR from my door."}


created_at: 2024-08-26 20:30:08
topic: persistent://default/default/dce8dee6-eb49-4277-b4b8-b3dbb5a5778f
id: 7412ea36-31c0-48aa-8eac-4870dfad9bd1
encoding: FLOAT32
metadata:
{"url": "https://huggingface.co/docs/peft/task_guides/prompt_based_methods", "title": "Prompt-based methods", "tokens": 3315, "retrieve_doc": true, "source": "peft", "_node_content": "{\"id_\": \"7412ea36-31c0-48aa-8eac-4870dfad9bd1\", \"embedding\": null, \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/prompt_based_methods\", \"title\": \"Prompt-based methods\", \"tokens\": 3315, \"retrieve_doc\": true, \"source\": \"peft\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d9e19246-3764-4463-993f-425191e1e412\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/prompt_based_methods\", \"title\": \"Prompt-based methods\", \"tokens\": 3315, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"a4974d540eeb655582d44b88d415425379b0d06f7d5e927999502cad9d07e2c7\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a7bec6ec-e4d9-4bb0-9fd6-3b7f0e9fc529\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://huggingface.co/docs/peft/task_guides/prompt_based_methods\", \"title\": \"Prompt-based methods\", \"tokens\": 3315, \"retrieve_doc\": true, \"source\": \"peft\"}, \"hash\": \"0880326e5e0a2477e8b33dc9ff1cffd922b5cb1df84a36b18a5ce1ce7406bd96\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11580, \"end_char_idx\": 13027, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d9e19246-3764-4463-993f-425191e1e412", "doc_id": "d9e19246-3764-4463-993f-425191e1e412", "ref_doc_id": "d9e19246-3764-4463-993f-425191e1e412", "chroma:document": "<div class=\"flex flex-col justify-center\">\n  <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/peft/PEFT-hub-screenshot.png\"/>\n  <figcaption class=\"text-center\">For example, the adapter weights for a opt-350m model stored on the Hub are only ~6MB compared to the full model size which can be ~700MB.</figcaption>\n</div>\n\n## Inference\n\nLet's load the model for inference and test it out on a tweet!\n\n```py\nfrom peft import AutoPeftModelForCausalLM\n\nmodel = AutoPeftModelForCausalLM.from_pretrained(\"peft_model_id\").to(\"cuda\")\ntokenizer = AutoTokenizer.from_pretrained(\"bigscience/bloomz-560m\")\n\ni = 15\ninputs = tokenizer(f'{text_column} : {ds[\"test\"][i][\"Tweet text\"]} Label : ', return_tensors=\"pt\")\nprint(ds[\"test\"][i][\"Tweet text\"])\n\"@NYTsupport i have complained a dozen times &amp; yet my papers are still thrown FAR from my door. Why is this so hard to resolve?\"\n```\n\nCall the [`~transformers.GenerationMixin.generate`] method to generate the predicted classification label.\n\n```py\nwith torch.no_grad():\n    inputs = {k: v.to(device) for k, v in inputs.items()}\n    outputs = model.generate(input_ids=inputs[\"input_ids\"], max_new_tokens=10)\n    print(tokenizer.batch_decode(outputs.detach().cpu().numpy(), skip_special_tokens=True))\n\"['Tweet text : @NYTsupport i have complained a dozen times &amp; yet my papers are still thrown FAR from my door. Why is this so hard to resolve? Label : complaint']\"\n```"}


