class_docstr: Implementation of the `RAdam optimizer`_.
  
  The learning rate warmup for Adam is a must-have trick for stable training in certain situations (or eps tuning).
  But the underlying mechanism is largely unknown. In our study, we suggest one fundamental cause is the large
  variance of the adaptive learning rates, and provide both theoretical and empirical support evidence.
  
  Args:
      params (iterable): iterable of parameters to optimize or dicts defining
          parameter groups
      lr (float, optional): learning rate (default: 1e-3)
      betas (Tuple[float, float], optional): coefficients used for computing
          running averages of gradient and its square (default: (0.9, 0.999))
      eps (float, optional): term added to the denominator to improve
          numerical stability (default: 1e-8)
      weight_decay (float, optional): weight decay coefficient (default: 0)
  
  .. _RAdam\: On the Variance of the Adaptive Learning Rate and Beyond:
      https://arxiv.org/abs/1908.03265
code: def __init__(
      self,
      params,
      lr=1e-3,
      betas=(0.9, 0.999),
      eps=1e-8,
      weight_decay=1e-4,
  ):
      defaults = dict(
          lr=lr, betas=betas, eps=eps, weight_decay=weight_decay
      )
      self.buffer = [[None, None, None] for ind in range(10)]
      super(RAdam, self).__init__(params, defaults)
  
imports: import math
  import torch
  from torch.optim.optimizer import Optimizer
label_desc: Use image node_id 1 to create a new RAdam object from inherited base classes: Optimizer with example: obj = RAdam(params, lr, betas, eps, weight_decay)