class_docstr: Modified from Fairseq and Use LightSeq adam kernel.
  
  Arguments:
      params (iterable): iterable of parameters to optimize or dicts defining
          parameter groups.
      lr (float, optional): learning rate. (default: 1e-3)
      betas (Tuple[float, float], optional): coefficients used for computing
          running averages of gradient and its square. (default: (0.9, 0.999))
      eps (float, optional): term added to the denominator to improve
          numerical stability. (default: 1e-8)
      weight_decay (float, optional): weight decay (L2 penalty) (default: 0)
      amsgrad (boolean, optional): whether to use the AMSGrad variant of this
          algorithm from the paper `On the Convergence of Adam and Beyond`_
          (default: False) NOT SUPPORTED in LSAdam!
      eps_inside_sqrt (boolean, optional): in the 'update parameters' step,
          adds eps to the bias-corrected second moment estimate before
          evaluating square root instead of adding it to the square root of
          second moment estimate as in the original paper. (default: False)
  .. _Adam: A Method for Stochastic Optimization:
      https://arxiv.org/abs/1412.6980
  .. _On the Convergence of Adam and Beyond:
      https://openreview.net/forum?id=ryQu7f-RZ
code: def __init__(
      self,
      params,
      lr=1e-3,
      bias_correction=True,
      betas=(0.9, 0.999),
      eps=1e-8,
      eps_inside_sqrt=False,
      weight_decay=0.0,
      max_grad_norm=0.0,
      amsgrad=False,
  ):
      global fused_adam_cuda
  
      if fused_adam_cuda is None:
          fused_adam_cuda = AdamBuilder().load()
  
      if amsgrad:
          raise RuntimeError(
              "LSAdam does not support the AMSGrad variant."
          )
      defaults = {
          "lr": lr,
          "bias_correction": bias_correction,
          "betas": betas,
          "eps": eps,
          "weight_decay": weight_decay,
          "max_grad_norm": max_grad_norm,
      }
      super().__init__(params, defaults)
      self.eps_mode = 0 if eps_inside_sqrt else 1
  
file_path: lightseq/lightseq/training/ops/pytorch/adam.py
imports: import torch
  from lightseq.training.ops.pytorch.builder import AdamBuilder
label_desc: Use image node_id 1 to create a new LSAdam object from inherited base classes: torch.optim with example: obj = LSAdam(params, lr, bias_correction, betas, eps, eps_inside_sqrt, weight_decay, max_grad_norm, amsgrad)