class_docstr: This module wraps a LLamaAttention module and injects adaption prompts.
docstr: Initialize object.
  
  Args:
      model_type: The transformer model type. This is used to retrieve the right method to
          compute query states.
      adapter_len: The length of the adaption prompt to insert.
      model: The original transformer attention module that is being wrapped.
code: def __init__(self, model_type: str, adapter_len: int, model):
      """
      Initialize object.
  
      Args:
          model_type: The transformer model type. This is used to retrieve the right method to
              compute query states.
          adapter_len: The length of the adaption prompt to insert.
          model: The original transformer attention module that is being wrapped.
      """
      assert not isinstance(model, AdaptedAttention)
      super().__init__()
      self.model_type = model_type
      self.model = model
      self.adapter_len = adapter_len
      # Assume all parameters of the attention model we are wrapping are on the same device.
      device = next(model.parameters()).device
      # Don't think this was specified in the paper, but we follow the official repo which used an Embedding
      # which initializes the tokens with standard normal values.
      # https://github.com/ZrrSkywalker/LLaMA-Adapter/blob/41c3546fe1997ab8a65809dc8d8f9252b19d9faf/llama/model.py#L234
      # (bsz, adapter_len, hidden_size)
      target_dtype = (
          model.q_proj.weight.dtype
          if model.q_proj.weight.dtype not in [torch.int8, torch.uint8]
          else torch.float32
      )
      self.adaption_prompt = nn.Parameter(
          torch.empty(
              1,
              adapter_len,
              self.model.hidden_size,
              device=device,
              dtype=target_dtype,
          ).normal_()
      )
      # Initialize the gate to 0 as this is "zero-init".
      self.adaption_gate = nn.Parameter(
          torch.zeros(1, device=device, dtype=target_dtype)
      )
  
file_path: peft/src/peft/tuners/adaption_prompt/layer.py
imports: import math
  import torch
  import torch.nn
  import torch.nn.functional
  from .config import TRANSFORMERS_MODEL_CONFIG
label_desc: Use image node_id 1 to create a new AdaptedAttention object from inherited base classes: nn with example: obj = AdaptedAttention(model_type, adapter_len, model)