class_docstr: Distributed optimizer, for all data types (fp16, bf16, and fp32).
  
  Arguments:
      optimizer: base optimizer such as Adam or SGD
      clip_grad: clip gradeints with this global L2 norm. Note
          that clipping is ignored if clip_grad == 0
      log_num_zeros_in_grad: return number of zeros in the gradients.
      check_for_nan_in_grad: check if gradients have a NaN.
      params_have_main_grad: flag indicating if parameters have
          a `main_grad` field. If this is set, we are assuming
          that the model parameters are store in the `main_grad`
          field instead of the typical `grad` field. This happens
          for the DDP cases where there is a continuous buffer
          holding the gradients. For example for bfloat16, we want
          to do gradient accumulation and all-reduces in float32
          and as a result we store those gradients in the main_grad.
          Note that main grad is not necessarily in float32.
      fp16: if true, the model is running in fp16.
      bf16: if true, the model is running in bfloat16.
      grad_scaler: used for scaling gradients. Note that this can be
          None. This case happens when `bf16 = True` and we don't
          use any loss scale. Note that for `bf16 = True`, we can have
          a constnat gradient scaler. Also for `bf16 = False`, we
          always require a grad scaler.
      models: list of models (i.e., the virtual pipelining models). This
          is used by the distributed optimizer for mapping parameters.
docstr: Waits on all_gather_handle if necessary, then copies params from param_buffer
  into model_params if necessary.
docstr_tok: ["Waits","on","all_gather_handle","if","necessary",",","then","copies","params","from","param_buffer","into","model_params","if","necessary","."]
code: def _finish_param_sync_helper(self, all_gather_handle_index):
      """
      Waits on all_gather_handle if necessary, then copies params from param_buffer
      into model_params if necessary.
      """
  
      # First check if there is an outstanding all-gather handle for this param.
      # If so, wait on the handle to ensure the communication is finished.
      if all_gather_handle_index >= len(self.all_gather_handles):
          return
  
      all_gather_handle = self.all_gather_handles[
          all_gather_handle_index
      ]
      if all_gather_handle is not None:
          all_gather_handle.wait()
          self.all_gather_handles[all_gather_handle_index] = None
  
          # Launch the all-gather for the next bucket now.
          # We can't pre-launch all-gathers for all buckets at once since we don't
          # want to head-of-line block the compute kernels with communication kernels
          # (since we run with CUDA_DEVICE_MAX_CONNECTIONS=1 to support sequence
          # parallelism).
          next_all_gather_handle_index = all_gather_handle_index + 1
          if next_all_gather_handle_index < self.num_all_gather_handles:
              self._dispatch_gather_model_params(
                  next_all_gather_handle_index
              )
  
      # Also check if we have already copied from the param buffer for this
      # handle; if not, complete the copy and mark as such.
      if not self.param_buffer_copied[all_gather_handle_index]:
          self._copy_params_from_param_buffer(all_gather_handle_index)
          self.param_buffer_copied[all_gather_handle_index] = True
  
imports: from apex.optimizers import FusedAdam
  import math
  import torch
  from megatron import get_args
  from megatron import get_timers
  from megatron import print_rank_0
  from megatron.core import mpu, tensor_parallel
  from .optimizer import MixedPrecisionOptimizer, _zero_grad_group_helper
  from .utils import shard_buffer
label_desc: Use image node_id 19 for calling the DistributedOptimizer obj's underlying member method code with example usage: obj._finish_param_sync_helper(all_gather_handle_index) without return types