class_docstr: This module wraps a LLamaAttention module and injects adaption prompts.
docstr: Forward pass for the adapter which wraps the original LlamaAttention module.
  
  "Official" paper implementation:
  https://github.com/ZrrSkywalker/LLaMA-Adapter/blob/41c3546fe1997ab8a65809dc8d8f9252b19d9faf/llama/model.py#L141
  
  Args:
      kwargs: See the original LlamaAttention module.
code: def forward(self, **kwargs):
      """
      Forward pass for the adapter which wraps the original LlamaAttention module.
  
      "Official" paper implementation:
      https://github.com/ZrrSkywalker/LLaMA-Adapter/blob/41c3546fe1997ab8a65809dc8d8f9252b19d9faf/llama/model.py#L141
  
      Args:
          kwargs: See the original LlamaAttention module.
      """
      if kwargs.get("output_attention", False):
          raise NotImplementedError(
              "output_attention is not currently supported."
          )
  
      output, _, past_key_value = self.model(**kwargs)
      bsz = output.shape[0]
      q_len = output.shape[1]
      embed_dim = output.shape[2]
      k_proj_layer = TRANSFORMERS_MODEL_CONFIG[
          self.model_type
      ].k_proj_layer
      v_proj_layer = TRANSFORMERS_MODEL_CONFIG[
          self.model_type
      ].v_proj_layer
      o_proj_layer = TRANSFORMERS_MODEL_CONFIG[
          self.model_type
      ].o_proj_layer
  
      if k_proj_layer == v_proj_layer:
          _, key, value = getattr(self.model, k_proj_layer)(
              self.adaption_prompt
          ).split(embed_dim, dim=2)
      else:
          key = getattr(self.model, k_proj_layer)(self.adaption_prompt)
          value = getattr(self.model, v_proj_layer)(
              self.adaption_prompt
          )
      # (bsz, num_heads, adapter_len, head_dim)
      adapter_k = (
          key.view(
              1,
              self.adapter_len,
              self.model.num_heads,
              self.model.head_dim,
          )
          .repeat(bsz, 1, 1, 1)
          .transpose(1, 2)
      )
      # (bsz, num_heads, adapter_len, head_dim)
      adapter_v = (
          value.view(
              1,
              self.adapter_len,
              self.model.num_heads,
              self.model.head_dim,
          )
          .repeat(bsz, 1, 1, 1)
          .transpose(1, 2)
      )
  
      # Recompute query states.
      compute_query_states = TRANSFORMERS_MODEL_CONFIG[
          self.model_type
      ].compute_query_states
      # (bsz, num_heads, q_len, head_dim)
      query_states = compute_query_states(model=self.model, **kwargs)
  
      previous_dtype = query_states.dtype
      # (bsz, num_heads, q_len, adapter_len)
      scores = torch.matmul(
          query_states, adapter_k.transpose(2, 3).to(previous_dtype)
      ) / math.sqrt(self.model.head_dim)
      # Upcast attention to fp32
      # (bsz, num_heads, q_len, adapter_len)
      scores = self.adaption_gate * F.softmax(
          scores, dim=-1, dtype=torch.float32
      ).to(previous_dtype)
      # (bsz, q_len, num_heads * head_dim)
      adapter_output = (
          torch.matmul(scores, adapter_v)
          .transpose(1, 2)
          .reshape(bsz, q_len, -1)
      )
      # (bsz, q_len, hidden_size)
      if o_proj_layer is not None:
          adapter_output = getattr(self.model, o_proj_layer)(
              adapter_output
          )
  
      # Add adaption prompt output to original output.
      output = output + adapter_output
  
      # Restore original dtype.
      output = output.to(previous_dtype)
      return output, None, past_key_value
  
file_path: peft/src/peft/tuners/adaption_prompt/layer.py
imports: import math
  import torch
  import torch.nn
  import torch.nn.functional
  from .config import TRANSFORMERS_MODEL_CONFIG
label_desc: Use image node_id 2 for calling the AdaptedAttention obj's underlying member method code with example usage: obj.forward() and returns: output,  None,  past_key_value