class_docstr: Distributed optimizer, for all data types (fp16, bf16, and fp32).
  
  Arguments:
      optimizer: base optimizer such as Adam or SGD
      clip_grad: clip gradeints with this global L2 norm. Note
          that clipping is ignored if clip_grad == 0
      log_num_zeros_in_grad: return number of zeros in the gradients.
      check_for_nan_in_grad: check if gradients have a NaN.
      params_have_main_grad: flag indicating if parameters have
          a `main_grad` field. If this is set, we are assuming
          that the model parameters are store in the `main_grad`
          field instead of the typical `grad` field. This happens
          for the DDP cases where there is a continuous buffer
          holding the gradients. For example for bfloat16, we want
          to do gradient accumulation and all-reduces in float32
          and as a result we store those gradients in the main_grad.
          Note that main grad is not necessarily in float32.
      fp16: if true, the model is running in fp16.
      bf16: if true, the model is running in bfloat16.
      grad_scaler: used for scaling gradients. Note that this can be
          None. This case happens when `bf16 = True` and we don't
          use any loss scale. Note that for `bf16 = True`, we can have
          a constnat gradient scaler. Also for `bf16 = False`, we
          always require a grad scaler.
      models: list of models (i.e., the virtual pipelining models). This
          is used by the distributed optimizer for mapping parameters.
docstr: Copy main params to model params.
  
  Since this step is followed by an all-gather through the DDP's grad
  buffer, this method is responsible for copying the updated params
  from the main shards into the correct position in the grad buffer.
docstr_tok: ["Copy","main","params","to","model","params",".","Since","this","step","is","followed","by","an","all-gather","through","the","DDP","'s","grad","buffer",",","this","method","is","responsible","for","copying","the","updated","params","from","the","main","shards","into","the","correct","position","in","the","grad","buffer","."]
code: def _copy_main_params_to_model_params(self):
      """
      Copy main params to model params.
  
      Since this step is followed by an all-gather through the DDP's grad
      buffer, this method is responsible for copying the updated params
      from the main shards into the correct position in the grad buffer.
      """
  
      # Utility method for copying group params.
      def copy_group_params(shard_main_groups, model_groups):
          for shard_main_group, model_group in zip(
              shard_main_groups, model_groups
          ):
              for shard_main_param, model_param in zip(
                  shard_main_group, model_group
              ):
                  param_range_map = self.get_model_param_range_map(
                      model_param
                  )
                  world_range = param_range_map["gbuf_world_in_bucket"]
  
                  assert world_range.size == shard_main_param.nelement()
  
                  (
                      model_id,
                      dtype,
                      bucket_id,
                  ) = self.model_param_gbuf_map[model_param]
                  model_param_buffer = self.param_buffers[model_id][
                      dtype
                  ][bucket_id]
  
                  shard_model_param = model_param_buffer.view(-1)[
                      world_range.start : world_range.end
                  ]
  
                  shard_model_param.data.copy_(shard_main_param)
  
      # Copy shard groups to model groups.
      copy_group_params(
          self.shard_fp32_from_float16_groups, self.model_float16_groups
      )
      copy_group_params(self.shard_fp32_groups, self.model_fp32_groups)
  
imports: from apex.optimizers import FusedAdam
  import math
  import torch
  from megatron import get_args
  from megatron import get_timers
  from megatron import print_rank_0
  from megatron.core import mpu, tensor_parallel
  from .optimizer import MixedPrecisionOptimizer, _zero_grad_group_helper
  from .utils import shard_buffer
label_desc: Use image node_id 24 for calling the DistributedOptimizer obj's underlying member method code with example usage: obj._copy_main_params_to_model_params() without return types