class_docstr: Implements Adam algorithm.
  
  Supports parameters updating on both GPU and CPU, depending on the device of parameters.
  But the parameters and gradients should on the same device:
    * Parameters on CPU and gradients on CPU is allowed.
    * Parameters on GPU and gradients on GPU is allowed.
    * Parameters on GPU and gradients on CPU is **not** allowed.
  
  `HybridAdam` requires CUDA extensions which can be built during installation or runtime.
  
  This version of Hybrid Adam is an hybrid of CPUAdam and FusedAdam.
  
  * For parameters updating on CPU, it uses CPUAdam.
  * For parameters updating on GPU, it uses FusedAdam.
  * Hybrid precision calculation of fp16 and fp32 is supported, eg fp32 parameters and fp16 gradients.
  
  :class:`colossalai.nn.optimizer.HybridAdam` may be used as a drop-in replacement for ``torch.optim.AdamW``,
  or ``torch.optim.Adam`` with ``adamw_mode=False``
  
  Adam was been proposed in `Adam: A Method for Stochastic Optimization`_.
  
  Arguments:
      model_params (iterable): iterable of parameters of dicts defining
          parameter groups.
      lr (float, optional): learning rate. (default: 1e-3)
      betas (Tuple[float, float], optional): coefficients used for computing
          running averages of gradient and its square. (default: (0.9, 0.999))
      eps (float, optional): term added to the denominator to improve
          numerical stability. (default: 1e-8)
      weight_decay (float, optional): weight decay (L2 penalty) (default: 0)
      amsgrad (boolean, optional): whether to use the AMSGrad variant of this
          algorithm from the paper `On the Convergence of Adam and Beyond`_
          (default: False) NOT SUPPORTED yet in CPUAdam!
      adamw_mode (boolean, optional): Apply L2 regularization or weight decay
          True for decoupled weight decay(also known as AdamW) (default: True)
      simd_log (boolean, optional): whether to show if you are using SIMD to
          accelerate. (default: False)
      nvme_offload_fraction (float, optional): Fraction of optimizer states to be offloaded to NVMe. Defaults to 0.0.
      nvme_offload_dir (Optional[str], optional): Directory to save NVMe offload files.
          If it's ``None``, a random temporary directory will be used. Defaults to None.
  
  .. _Adam\: A Method for Stochastic Optimization:
      https://arxiv.org/abs/1412.6980
  .. _On the Convergence of Adam and Beyond:
      https://openreview.net/forum?id=ryQu7f-RZ
code: def __init__(
      self,
      model_params,
      lr=1e-3,
      bias_correction=True,
      betas=(0.9, 0.999),
      eps=1e-8,
      weight_decay=0,
      adamw_mode=True,
      nvme_offload_fraction: float = 0.0,
      nvme_offload_dir: Optional[str] = None,
      **defaults: Any,
  ):
      super().__init__(
          model_params,
          lr,
          bias_correction,
          betas,
          eps,
          weight_decay,
          adamw_mode,
          nvme_offload_fraction,
          nvme_offload_dir,
      )
      if torch.cuda.is_available():
          fused_optim = FusedOptimBuilder().load()
          self.gpu_adam_op = fused_optim.multi_tensor_adam
          self._dummy_overflow_buf = torch.cuda.IntTensor([0])
  
file_path: ColossalAI/colossalai/nn/optimizer/hybrid_adam.py
imports: from typing import Any, Optional
  import torch
  from colossalai.kernel.op_builder import FusedOptimBuilder
  from colossalai.utils import multi_tensor_applier
  from .cpu_adam import CPUAdam
label_desc: Use image node_id 1 to create a new HybridAdam object from inherited base classes: CPUAdam with example: obj = HybridAdam(model_params, lr, bias_correction, betas, eps, weight_decay, adamw_mode, nvme_offload_fraction, nvme_offload_dir)