class_docstr: A class containing all functions for loading and using adapters weights that are supported in PEFT library. For
  more details about adapters and injecting them on a transformer-based model, check out the documentation of PEFT
  library: https://huggingface.co/docs/peft/index
  
  Currently supported PEFT methods are all non-prefix tuning methods. Below is the list of supported PEFT methods
  that anyone can load, train and run with this mixin class:
  - Low Rank Adapters (LoRA): https://huggingface.co/docs/peft/conceptual_guides/lora
  - IA3: https://huggingface.co/docs/peft/conceptual_guides/ia3
  - AdaLora: https://arxiv.org/abs/2303.10512
  
  Other PEFT models such as prompt tuning, prompt learning are out of scope as these adapters are not "injectable"
  into a torch module. For using these methods, please refer to the usage guide of PEFT library.
  
  With this mixin, if the correct PEFT version is installed, it is possible to:
  
  - Load an adapter stored on a local path or in a remote Hub repository, and inject it in the model
  - Attach new adapters in the model and train them with Trainer or by your own.
  - Attach multiple adapters and iteratively activate / deactivate them
  - Activate / deactivate all adapters from the model.
  - Get the `state_dict` of the active adapter.
class_docstr_tok: ["A","class","containing","all","functions","for","loading","and","using","adapters","weights","that","are","supported","in","PEFT","library",".","For","more","details","about","adapters","and","injecting","them","on","a","transformer-based","model",",","check","out","the","documentation","of","PEFT","library",":","https",":","\/\/huggingface.co\/docs\/peft\/index","Currently","supported","PEFT","methods","are","all","non-prefix","tuning","methods",".","Below","is","the","list","of","supported","PEFT","methods","that","anyone","can","load",",","train","and","run","with","this","mixin","class",":","-","Low","Rank","Adapters","(","LoRA",")",":","https",":","\/\/huggingface.co\/docs\/peft\/conceptual_guides\/lora","-","IA3",":","https",":","\/\/huggingface.co\/docs\/peft\/conceptual_guides\/ia3","-","AdaLora",":","https",":","\/\/arxiv.org\/abs\/2303.10512","Other","PEFT","models","such","as","prompt","tuning",",","prompt","learning","are","out","of","scope","as","these","adapters","are","not","``","injectable","''","into","a","torch","module",".","For","using","these","methods",",","please","refer","to","the","usage","guide","of","PEFT","library",".","With","this","mixin",",","if","the","correct","PEFT","version","is","installed",",","it","is","possible","to",":","-","Load","an","adapter","stored","on","a","local","path","or","in","a","remote","Hub","repository",",","and","inject","it","in","the","model","-","Attach","new","adapters","in","the","model","and","train","them","with","Trainer","or","by","your","own",".","-","Attach","multiple","adapters","and","iteratively","activate","\/","deactivate","them","-","Activate","\/","deactivate","all","adapters","from","the","model",".","-","Get","the","`","state_dict","`","of","the","active","adapter","."]
docstr: Load adapter weights from file or remote Hub folder. If you are not familiar with adapters and PEFT methods, we
  invite you to read more about them on PEFT official documentation: https://huggingface.co/docs/peft
  
  Requires peft as a backend to load the adapter weights.
  
  Args:
      peft_model_id (`str`, *optional*):
          The identifier of the model to look for on the Hub, or a local path to the saved adapter config file
          and adapter weights.
      adapter_name (`str`, *optional*):
          The adapter name to use. If not set, will use the default adapter.
      revision (`str`, *optional*, defaults to `"main"`):
          The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a
          git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any
          identifier allowed by git.
  
          <Tip>
  
          To test a pull request you made on the Hub, you can pass `revision="refs/pr/<pr_number>".
  
          </Tip>
  
      token (`str`, `optional`):
          Whether to use authentication token to load the remote folder. Userful to load private repositories
          that are on HuggingFace Hub. You might need to call `huggingface-cli login` and paste your tokens to
          cache it.
      device_map (`str` or `Dict[str, Union[int, str, torch.device]]` or `int` or `torch.device`, *optional*):
          A map that specifies where each submodule should go. It doesn't need to be refined to each
          parameter/buffer name, once a given module name is inside, every submodule of it will be sent to the
          same device. If we only pass the device (*e.g.*, `"cpu"`, `"cuda:1"`, `"mps"`, or a GPU ordinal rank
          like `1`) on which the model will be allocated, the device map will map the entire model to this
          device. Passing `device_map = 0` means put the whole model on GPU 0.
  
          To have Accelerate compute the most optimized `device_map` automatically, set `device_map="auto"`. For
          more information about each option see [designing a device
          map](https://hf.co/docs/accelerate/main/en/usage_guides/big_modeling#designing-a-device-map).
      max_memory (`Dict`, *optional*):
          A dictionary device identifier to maximum memory. Will default to the maximum memory available for each
          GPU and the available CPU RAM if unset.
      offload_folder (`str` or `os.PathLike`, `optional`):
          If the `device_map` contains any value `"disk"`, the folder where we will offload weights.
      offload_index (`int`, `optional`):
          `offload_index` argument to be passed to `accelerate.dispatch_model` method.
      peft_config (`Dict[str, Any]`, *optional*):
          The configuration of the adapter to add, supported adapters are non-prefix tuning and adaption prompts
          methods. This argument is used in case users directly pass PEFT state dicts
      adapter_state_dict (`Dict[str, torch.Tensor]`, *optional*):
          The state dict of the adapter to load. This argument is used in case users directly pass PEFT state
          dicts
      adapter_kwargs (`Dict[str, Any]`, *optional*):
          Additional keyword arguments passed along to the `from_pretrained` method of the adapter config and
          `find_adapter_config_file` method.
docstr_tok: ["Load","adapter","weights","from","file","or","remote","Hub","folder",".","If","you","are","not","familiar","with","adapters","and","PEFT","methods",",","we","invite","you","to","read","more","about","them","on","PEFT","official","documentation",":","https",":","\/\/huggingface.co\/docs\/peft","Requires","peft","as","a","backend","to","load","the","adapter","weights",".","Args",":","peft_model_id","(","`","str","`",",","*","optional","*",")",":","The","identifier","of","the","model","to","look","for","on","the","Hub",",","or","a","local","path","to","the","saved","adapter","config","file","and","adapter","weights",".","adapter_name","(","`","str","`",",","*","optional","*",")",":","The","adapter","name","to","use",".","If","not","set",",","will","use","the","default","adapter",".","revision","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","main","''","`",")",":","The","specific","model","version","to","use",".","It","can","be","a","branch","name",",","a","tag","name",",","or","a","commit","id",",","since","we","use","a","git-based","system","for","storing","models","and","other","artifacts","on","huggingface.co",",","so","`","revision","`","can","be","any","identifier","allowed","by","git",".","<","Tip",">","To","test","a","pull","request","you","made","on","the","Hub",",","you","can","pass","`","revision=","''","refs\/pr\/","<","pr_number",">","''",".","<","\/Tip",">","token","(","`","str","`",",","`","optional","`",")",":","Whether","to","use","authentication","token","to","load","the","remote","folder",".","Userful","to","load","private","repositories","that","are","on","HuggingFace","Hub",".","You","might","need","to","call","`","huggingface-cli","login","`","and","paste","your","tokens","to","cache","it",".","device_map","(","`","str","`","or","`","Dict","[","str",",","Union","[","int",",","str",",","torch.device","]","]","`","or","`","int","`","or","`","torch.device","`",",","*","optional","*",")",":","A","map","that","specifies","where","each","submodule","should","go",".","It","does","n't","need","to","be","refined","to","each","parameter\/buffer","name",",","once","a","given","module","name","is","inside",",","every","submodule","of","it","will","be","sent","to","the","same","device",".","If","we","only","pass","the","device","(","*","e.g",".","*",",","`","``","cpu","''","`",",","`","``","cuda:1","''","`",",","`","``","mps","''","`",",","or","a","GPU","ordinal","rank","like","`","1","`",")","on","which","the","model","will","be","allocated",",","the","device","map","will","map","the","entire","model","to","this","device",".","Passing","`","device_map","=","0","`","means","put","the","whole","model","on","GPU","0",".","To","have","Accelerate","compute","the","most","optimized","`","device_map","`","automatically",",","set","`","device_map=","''","auto","''","`",".","For","more","information","about","each","option","see","[","designing","a","device","map","]","(","https",":","\/\/hf.co\/docs\/accelerate\/main\/en\/usage_guides\/big_modeling","#","designing-a-device-map",")",".","max_memory","(","`","Dict","`",",","*","optional","*",")",":","A","dictionary","device","identifier","to","maximum","memory",".","Will","default","to","the","maximum","memory","available","for","each","GPU","and","the","available","CPU","RAM","if","unset",".","offload_folder","(","`","str","`","or","`","os.PathLike","`",",","`","optional","`",")",":","If","the","`","device_map","`","contains","any","value","`","``","disk","''","`",",","the","folder","where","we","will","offload","weights",".","offload_index","(","`","int","`",",","`","optional","`",")",":","`","offload_index","`","argument","to","be","passed","to","`","accelerate.dispatch_model","`","method",".","peft_config","(","`","Dict","[","str",",","Any","]","`",",","*","optional","*",")",":","The","configuration","of","the","adapter","to","add",",","supported","adapters","are","non-prefix","tuning","and","adaption","prompts","methods",".","This","argument","is","used","in","case","users","directly","pass","PEFT","state","dicts","adapter_state_dict","(","`","Dict","[","str",",","torch.Tensor","]","`",",","*","optional","*",")",":","The","state","dict","of","the","adapter","to","load",".","This","argument","is","used","in","case","users","directly","pass","PEFT","state","dicts","adapter_kwargs","(","`","Dict","[","str",",","Any","]","`",",","*","optional","*",")",":","Additional","keyword","arguments","passed","along","to","the","`","from_pretrained","`","method","of","the","adapter","config","and","`","find_adapter_config_file","`","method","."]
code: def load_adapter(
      self,
      peft_model_id: Optional[str] = None,
      adapter_name: Optional[str] = None,
      revision: Optional[str] = None,
      token: Optional[str] = None,
      device_map: Optional[str] = "auto",
      max_memory: Optional[str] = None,
      offload_folder: Optional[str] = None,
      offload_index: Optional[int] = None,
      peft_config: Dict[str, Any] = None,
      adapter_state_dict: Optional[Dict[str, "torch.Tensor"]] = None,
      adapter_kwargs: Optional[Dict[str, Any]] = None,
  ) -> None:
      """
      Load adapter weights from file or remote Hub folder. If you are not familiar with adapters and PEFT methods, we
      invite you to read more about them on PEFT official documentation: https://huggingface.co/docs/peft
  
      Requires peft as a backend to load the adapter weights.
  
      Args:
          peft_model_id (`str`, *optional*):
              The identifier of the model to look for on the Hub, or a local path to the saved adapter config file
              and adapter weights.
          adapter_name (`str`, *optional*):
              The adapter name to use. If not set, will use the default adapter.
          revision (`str`, *optional*, defaults to `"main"`):
              The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a
              git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any
              identifier allowed by git.
  
              <Tip>
  
              To test a pull request you made on the Hub, you can pass `revision="refs/pr/<pr_number>".
  
              </Tip>
  
          token (`str`, `optional`):
              Whether to use authentication token to load the remote folder. Userful to load private repositories
              that are on HuggingFace Hub. You might need to call `huggingface-cli login` and paste your tokens to
              cache it.
          device_map (`str` or `Dict[str, Union[int, str, torch.device]]` or `int` or `torch.device`, *optional*):
              A map that specifies where each submodule should go. It doesn't need to be refined to each
              parameter/buffer name, once a given module name is inside, every submodule of it will be sent to the
              same device. If we only pass the device (*e.g.*, `"cpu"`, `"cuda:1"`, `"mps"`, or a GPU ordinal rank
              like `1`) on which the model will be allocated, the device map will map the entire model to this
              device. Passing `device_map = 0` means put the whole model on GPU 0.
  
              To have Accelerate compute the most optimized `device_map` automatically, set `device_map="auto"`. For
              more information about each option see [designing a device
              map](https://hf.co/docs/accelerate/main/en/usage_guides/big_modeling#designing-a-device-map).
          max_memory (`Dict`, *optional*):
              A dictionary device identifier to maximum memory. Will default to the maximum memory available for each
              GPU and the available CPU RAM if unset.
          offload_folder (`str` or `os.PathLike`, `optional`):
              If the `device_map` contains any value `"disk"`, the folder where we will offload weights.
          offload_index (`int`, `optional`):
              `offload_index` argument to be passed to `accelerate.dispatch_model` method.
          peft_config (`Dict[str, Any]`, *optional*):
              The configuration of the adapter to add, supported adapters are non-prefix tuning and adaption prompts
              methods. This argument is used in case users directly pass PEFT state dicts
          adapter_state_dict (`Dict[str, torch.Tensor]`, *optional*):
              The state dict of the adapter to load. This argument is used in case users directly pass PEFT state
              dicts
          adapter_kwargs (`Dict[str, Any]`, *optional*):
              Additional keyword arguments passed along to the `from_pretrained` method of the adapter config and
              `find_adapter_config_file` method.
      """
      check_peft_version(min_version=MIN_PEFT_VERSION)
  
      adapter_name = (
          adapter_name if adapter_name is not None else "default"
      )
      if adapter_kwargs is None:
          adapter_kwargs = {}
  
      from peft import (
          PeftConfig,
          inject_adapter_in_model,
          load_peft_weights,
      )
      from peft.utils import set_peft_model_state_dict
  
      if (
          self._hf_peft_config_loaded
          and adapter_name in self.peft_config
      ):
          raise ValueError(
              f"Adapter with name {adapter_name} already exists. Please use a different name."
          )
  
      if peft_model_id is None and (
          adapter_state_dict is None and peft_config is None
      ):
          raise ValueError(
              "You should either pass a `peft_model_id` or a `peft_config` and `adapter_state_dict` to load an adapter."
          )
  
      # We keep `revision` in the signature for backward compatibility
      if revision is not None and "revision" not in adapter_kwargs:
          adapter_kwargs["revision"] = revision
      elif (
          revision is not None
          and "revision" in adapter_kwargs
          and revision != adapter_kwargs["revision"]
      ):
          logger.error(
              "You passed a `revision` argument both in `adapter_kwargs` and as a standalone argument. "
              "The one in `adapter_kwargs` will be used."
          )
  
      # Override token with adapter_kwargs' token
      if "token" in adapter_kwargs:
          token = adapter_kwargs.pop("token")
  
      if peft_config is None:
          adapter_config_file = find_adapter_config_file(
              peft_model_id,
              token=token,
              **adapter_kwargs,
          )
  
          if adapter_config_file is None:
              raise ValueError(
                  f"adapter model file not found in {peft_model_id}. Make sure you are passing the correct path to the "
                  "adapter model."
              )
  
          peft_config = PeftConfig.from_pretrained(
              peft_model_id,
              token=token,
              **adapter_kwargs,
          )
  
      # Create and add fresh new adapters into the model.
      inject_adapter_in_model(peft_config, self, adapter_name)
  
      if not self._hf_peft_config_loaded:
          self._hf_peft_config_loaded = True
  
      if peft_model_id is not None:
          adapter_state_dict = load_peft_weights(
              peft_model_id, token=token, **adapter_kwargs
          )
  
      # We need to pre-process the state dict to remove unneeded prefixes - for backward compatibility
      processed_adapter_state_dict = {}
      prefix = "base_model.model."
      for key, value in adapter_state_dict.items():
          if key.startswith(prefix):
              new_key = key[len(prefix) :]
          else:
              new_key = key
          processed_adapter_state_dict[new_key] = value
  
      # Load state dict
      incompatible_keys = set_peft_model_state_dict(
          self, processed_adapter_state_dict, adapter_name
      )
  
      if incompatible_keys is not None:
          # check only for unexpected keys
          if (
              hasattr(incompatible_keys, "unexpected_keys")
              and len(incompatible_keys.unexpected_keys) > 0
          ):
              logger.warning(
                  f"Loading adapter weights from {peft_model_id} led to unexpected keys not found in the model: "
                  f" {incompatible_keys.unexpected_keys}. "
              )
  
      # Re-dispatch model and hooks in case the model is offloaded to CPU / Disk.
      if (
          (getattr(self, "hf_device_map", None) is not None)
          and (
              len(
                  set(self.hf_device_map.values()).intersection(
                      {"cpu", "disk"}
                  )
              )
              > 0
          )
          and len(self.peft_config) == 1
      ):
          self._dispatch_accelerate_model(
              device_map=device_map,
              max_memory=max_memory,
              offload_folder=offload_folder,
              offload_index=offload_index,
          )
  
file_path: transformers/src/transformers/integrations/peft.py
imports: import inspect
  import warnings
  from typing import TYPE_CHECKING, Any, Dict, List, Optional, Union
  from ..utils import check_peft_version, find_adapter_config_file, is_accelerate_available, is_peft_available, is_torch_available, logging
label_desc: Use image node_id 1 for calling the PeftAdapterMixin obj's underlying member method code with example usage: obj.load_adapter(peft_model_id, adapter_name, revision, token, device_map, max_memory, offload_folder, offload_index, peft_config, adapter_state_dict, adapter_kwargs) without return types