class_docstr: A wrapper that is used to replace the DDP module provided by Megatron, which
  is adapted to enable the sophiscated parallel and reduction strategies in
  Fast MoE.
code: def __init__(
      self,
      module,
      accumulate_allreduce_grads_in_fp32=False,
      use_contiguous_buffers_in_ddp=False,
  ):
      assert (
          not accumulate_allreduce_grads_in_fp32
      ), "FastMoE not supports accumulate_allrecude_grads_in_fp32"
      assert (
          not use_contiguous_buffers_in_ddp
      ), "FastMoE not supports use_contiguous_buffers_in_ddp"
  
      if _groups is None:
          _init()
      super().__init__(module, **_groups)
  
file_path: smart-moe/fmoe/megatron/distributed.py
imports: import torch
  from fmoe.distributed import DistributedGroupedDataParallel
label_desc: Use image node_id 1 to create a new DistributedDataParallel object from inherited base classes: DistributedGroupedDataParallel with example: obj = DistributedDataParallel(module, accumulate_allreduce_grads_in_fp32, use_contiguous_buffers_in_ddp)