class_docstr: LoRA where single nn.Linear represents more than one layer (used in some implementations of attention query/key/value projections). Weights of linear layer are set to be frozen per default.
  
  Parameters
  ----------
  in_features
      input dimension, set to the original linear layer input dimension LoRA is replacing
  out_features
      output dimension, set to the original linear layer output dimension LoRA is replacing
  r
      rank r of the low-rank decomposition
  lora_alpha
      Scaling factor. Can be simply set to same value as r as initialization is scaled already.
  lora_dropout
      Dropout rate for LoRA
  fan_in_fan_out
      Set this to True if the layer to replace stores weight like (fan_in, fan_out)
  merge_weights
      Merging weights during inference to reduce latency
  
  References
  ----------
  1. Edward J. Hu*, Yelong Shen*, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen,
  "LoRA: Low-Rank Adaptation of Large Language Models", 2021
  https://arxiv.org/abs/2106.09685
  2. Code: https://github.com/microsoft/LoRA
code: def __init__(
      self,
      in_features: int,
      out_features: int,
      r: int = 0,
      lora_alpha: int = 1,
      lora_dropout: float = 0.0,
      enable_lora: List[bool] = [False],
      fan_in_fan_out: bool = False,
      merge_weights: bool = True,
      **kwargs,
  ):
      nn.Linear.__init__(self, in_features, out_features, **kwargs)
      LoRALayer.__init__(
          self,
          r=r,
          lora_alpha=lora_alpha,
          lora_dropout=lora_dropout,
          merge_weights=merge_weights,
      )
      assert (
          out_features % len(enable_lora) == 0
      ), "The length of enable_lora must divide out_features"
      self.enable_lora = enable_lora
      self.fan_in_fan_out = fan_in_fan_out
      # Actual trainable parameters
      if r > 0 and any(enable_lora):
          self.lora_A = nn.Parameter(
              self.weight.new_zeros((r * sum(enable_lora), in_features))
          )
          self.lora_B = nn.Parameter(
              self.weight.new_zeros(
                  (
                      out_features
                      // len(enable_lora)
                      * sum(enable_lora),
                      r,
                  )
              )
          )  # weights for Conv1D with groups=sum(enable_lora)
          self.scaling = self.lora_alpha / self.r
          # Freezing the pre-trained weight matrix
          self.weight.requires_grad = False
          # Compute the indices
          self.lora_ind = self.weight.new_zeros(
              (out_features,), dtype=torch.bool
          ).view(len(enable_lora), -1)
          self.lora_ind[enable_lora, :] = True
          self.lora_ind = self.lora_ind.view(-1)
      self.reset_parameters()
      if fan_in_fan_out:
          self.weight.data = self.weight.data.T
  
imports: import math
  from typing import List
  import torch
  import torch.nn
  import torch.nn.functional
label_desc: Use image node_id 1 to create a new LoRAMergedLinear object from inherited base classes: nn, LoRALayer with example: obj = LoRAMergedLinear(in_features, out_features, r, lora_alpha, lora_dropout, enable_lora, fan_in_fan_out, merge_weights)