class_docstr: Float16 optimizer wrapper for half precision (fp16 and bf16) data types.
  This optimizer wrapper holds main parameters and gradients in fp32 to support
  stable convergence.
  
  Arguments:
      optimizer: base optimizer such as Adam or SGD.
      fp32_grad_accum: to enable the use of fp32 in gradient accumulation and allreduce.
      contiguous_grad_bucket: to enable allocating the master gradients in the 
          contiguous memory space to reduce memory fragmentation.
      async_grad_allreduce: enable asynchronous gradient allreduce that is executed
          along with the training step backprop.
code: def step(self, **kwargs):
      # while async grad allreduce is enabled, bprop will keep moving forward without waiting for
      # the finish of async grad AR works. Hence, to guarantee the correctness of grads reduction,
      # we cannot start weight update until all async grad AR works are done.
      if self._async_grad_allreduce:
          torch.cuda.synchronize()
  
      # Step the optimizer.
      self.optimizer.step(closure=None, **kwargs)
  
      # Update params from main params.
      with torch.no_grad():
          self._copy_main_params_to_model_params()
  
      # Successful update.
      return True
  
imports: from contextlib import contextmanager
  import torch
  from nemo.utils import logging
label_desc: Use image node_id 10 for calling the MainParamsOptimizerWrapper obj's underlying member method code with example usage: obj.step() and returns: True