class_docstr: LoRA where single nn.Linear represents more than one layer (used in some implementations of attention query/key/value projections). Weights of linear layer are set to be frozen per default.
  
  Parameters
  ----------
  in_features
      input dimension, set to the original linear layer input dimension LoRA is replacing
  out_features
      output dimension, set to the original linear layer output dimension LoRA is replacing
  r
      rank r of the low-rank decomposition
  lora_alpha
      Scaling factor. Can be simply set to same value as r as initialization is scaled already.
  lora_dropout
      Dropout rate for LoRA
  fan_in_fan_out
      Set this to True if the layer to replace stores weight like (fan_in, fan_out)
  merge_weights
      Merging weights during inference to reduce latency
  
  References
  ----------
  1. Edward J. Hu*, Yelong Shen*, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen,
  "LoRA: Low-Rank Adaptation of Large Language Models", 2021
  https://arxiv.org/abs/2106.09685
  2. Code: https://github.com/microsoft/LoRA
code: def forward(self, x: torch.Tensor):
      def T(w):
          return w.T if self.fan_in_fan_out else w
  
      if self.merged:
          return F.linear(x, T(self.weight), bias=self.bias)
      else:
          result = F.linear(x, T(self.weight), bias=self.bias)
          if self.r > 0:
              after_A = F.linear(self.lora_dropout(x), self.lora_A)
              after_B = F.conv1d(
                  after_A.transpose(-2, -1),
                  self.lora_B.unsqueeze(-1),
                  groups=sum(self.enable_lora),
              ).transpose(-2, -1)
              result += self.zero_pad(after_B) * self.scaling
          return result
  
imports: import math
  from typing import List
  import torch
  import torch.nn
  import torch.nn.functional
label_desc: Use image node_id 6 for calling the LoRAMergedLinear obj's underlying member method code with example usage: obj.forward(x) and returns: unknown, F, result