class_docstr: Abstract LoRA Layer.
  
  Parameters
  ----------
  r
      rank r of the low-rank decomposition.
  lora_alpha
      Scaling factor. Can be simply set to same value as r as initialization is scaled already.
  merge_weights
      Merging weights during inference to reduce latency.
  
  References
  ----------
  1. Edward J. Hu*, Yelong Shen*, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen,
  "LoRA: Low-Rank Adaptation of Large Language Models", 2021
  https://arxiv.org/abs/2106.09685
  2. Code: https://github.com/microsoft/LoRA
code: def __init__(
      self,
      r: int,
      lora_alpha: int,
      lora_dropout: float,
      merge_weights: bool,
  ):
      self.r = r
      self.lora_alpha = lora_alpha
      # Optional dropout
      if lora_dropout > 0.0:
          self.lora_dropout = nn.Dropout(p=lora_dropout)
      else:
          self.lora_dropout = identity
      # Mark the weight as unmerged
      self.merged = False
      self.merge_weights = merge_weights
  
imports: import math
  from typing import List
  import torch
  import torch.nn
  import torch.nn.functional
label_desc: Use image node_id 1 to create a new LoRALayer object  with example: obj = LoRALayer(r, lora_alpha, lora_dropout, merge_weights)