class_docstr: Implements BERT version of Adam algorithm with weight decay fix.
  Params:
      lr: learning rate
      warmup: portion of t_total for the warmup, -1 means no warmup. Default: -1
      t_total: total number of training steps for the learning
          rate schedule, -1  means constant learning rate of 1. (no warmup regardless of warmup setting). Default: -1
      schedule: schedule to use for the warmup (see above).
          Can be `'warmup_linear'`, `'warmup_constant'`, `'warmup_cosine'`, `'none'`, `None` or a `_LRSchedule` object (see below).
          If `None` or `'none'`, learning rate is always kept constant.
          Default : `'warmup_linear'`
      b1: Adams b1. Default: 0.9
      b2: Adams b2. Default: 0.999
      e: Adams epsilon. Default: 1e-6
      weight_decay: Weight decay. Default: 0.01
      max_grad_norm: Maximum norm for the gradients (-1 means no clipping). Default: 1.0
docstr: Performs a single optimization step.
  
  Arguments:
      closure (callable, optional): A closure that reevaluates the model
          and returns the loss.
code: def step(self, closure=None):
      """Performs a single optimization step.
  
      Arguments:
          closure (callable, optional): A closure that reevaluates the model
              and returns the loss.
      """
      loss = None
      if closure is not None:
          loss = closure()
  
      for group in self.param_groups:
          for p in group["params"]:
              if p.grad is None:
                  continue
              grad = p.grad.data
              if grad.is_sparse:
                  raise RuntimeError(
                      "Adam does not support sparse gradients, please consider SparseAdam instead"
                  )
  
              state = self.state[p]
  
              # State initialization
              if len(state) == 0:
                  state["step"] = 0
                  # Exponential moving average of gradient values
                  state["next_m"] = torch.zeros_like(p.data)
                  # Exponential moving average of squared gradient values
                  state["next_v"] = torch.zeros_like(p.data)
  
              next_m, next_v = state["next_m"], state["next_v"]
              beta1, beta2 = group["b1"], group["b2"]
  
              # Add grad clipping
              if group["max_grad_norm"] > 0:
                  clip_grad_norm_(p, group["max_grad_norm"])
  
              # Decay the first and second moment running average coefficient
              # In-place operations to update the averages at the same time
              next_m.mul_(beta1).add_(alpha=1 - beta1, other=grad)
              next_v.mul_(beta2).addcmul_(
                  value=1 - beta2, tensor1=grad, tensor2=grad
              )
              update = next_m / (next_v.sqrt() + group["e"])
  
              # Just adding the square of the weights to the loss function is *not*
              # the correct way of using L2 regularization/weight decay with Adam,
              # since that will interact with the m and v parameters in strange ways.
              #
              # Instead we want to decay the weights in a manner that doesn't interact
              # with the m/v parameters. This is equivalent to adding the square
              # of the weights to the loss with plain (non-momentum) SGD.
              if group["weight_decay"] > 0.0:
                  update += group["weight_decay"] * p.data
  
              lr_scheduled = group["lr"]
              lr_scheduled *= group["schedule"].get_lr(state["step"])
  
              update_with_lr = lr_scheduled * update
              p.data.add_(-update_with_lr)
  
              state["step"] += 1
  
              # step_size = lr_scheduled * math.sqrt(bias_correction2) / bias_correction1
              # No bias correction
              # bias_correction1 = 1 - beta1 ** state['step']
              # bias_correction2 = 1 - beta2 ** state['step']
  
      return loss
  
imports: import math
  import torch
  from torch.optim.optimizer import Optimizer
  from torch.nn.utils import clip_grad_norm_
  import logging
  import abc
  import sys
label_desc: Use image node_id 3 for calling the BertAdam obj's underlying member method code with example usage: obj.step(closure) and returns: loss