class_docstr: This AdamW implementation is copied from Huggingface.
  We modified it with Adagrad scaling by rms of a weight tensor
  Implements Adam algorithm with weight decay fix as introduced in [Decoupled Weight Decay
  Regularization](https://arxiv.org/abs/1711.05101).
  Parameters:
      params (`Iterable[nn.parameter.Parameter]`):
          Iterable of parameters to optimize or dictionaries defining parameter groups.
      lr (`float`, *optional*, defaults to 1e-3):
          The learning rate to use.
      betas (`Tuple[float,float]`, *optional*, defaults to (0.9, 0.999)):
          Adam's betas parameters (b1, b2).
      eps (`float`, *optional*, defaults to 1e-6):
          Adam's epsilon for numerical stability.
      weight_decay (`float`, *optional*, defaults to 0):
          Decoupled weight decay to apply.
      correct_bias (`bool`, *optional*, defaults to `True`):
          Whether or not to correct bias in Adam (for instance, in Bert TF repository they use `False`).
      no_deprecation_warning (`bool`, *optional*, defaults to `False`):
          A flag used to disable the deprecation warning (set to `True` to disable the warning).
code: def __init__(
      self,
      params: Iterable[nn.parameter.Parameter],
      lr: float = 1e-3,
      betas: Tuple[float, float] = (0.9, 0.999),
      eps: float = 1e-6,
      weight_decay: float = 0.0,
      correct_bias: bool = True,
  ):
      if lr < 0.0:
          raise ValueError(
              f"Invalid learning rate: {lr} - should be >= 0.0"
          )
      if not 0.0 <= betas[0] < 1.0:
          raise ValueError(
              f"Invalid beta parameter: {betas[0]} - should be in [0.0, 1.0)"
          )
      if not 0.0 <= betas[1] < 1.0:
          raise ValueError(
              f"Invalid beta parameter: {betas[1]} - should be in [0.0, 1.0)"
          )
      if not 0.0 <= eps:
          raise ValueError(
              f"Invalid epsilon value: {eps} - should be >= 0.0"
          )
      defaults = dict(
          lr=lr,
          betas=betas,
          eps=eps,
          weight_decay=weight_decay,
          correct_bias=correct_bias,
      )
      super().__init__(params, defaults)
  
file_path: bert-ultrafast/training/cramming/backend/optimizers/nanoT5_optimizer.py
imports: import torch
  import math
  from typing import Iterable, Tuple
  from torch import nn
label_desc: Use image node_id 1 to create a new AdamWScale object from inherited base classes: torch.optim with example: obj = AdamWScale(params, lr, betas, eps, weight_decay, correct_bias)