class_docstr: A customized DDP module to support different all-reduce regions in the
  model.  The all-reduce region is defined as an attribution `dp_comm` in the
  weight object.
  The grads of the weights are identified to be reduced in different groups
  according to the weigths' `dp_comm` attribute.
  If it is set to `dp`, it will only be reduced across the data-parallel
  groups, which means that in the model parallel group, they are not
  synchronized.
  If it is set to `world`, the gradients is synchronized across all workers,
  regardless their model or data parallel group. This is extremely useful for
  shared layers like the gate.
docstr: Directly call the module's forward function.
docstr_tok: ["Directly","call","the","module","'s","forward","function","."]
code: def forward(self, *args, **kwargs):
      r"""
      Directly call the module's forward function.
      """
      return self.module(*args, **kwargs)
  
file_path: smart-moe/fmoe/distributed.py
imports: import torch
  import torch.nn
  from torch._utils import _flatten_dense_tensors, _unflatten_dense_tensors
  from .utils import get_torch_default_comm, get_rank_0_in_comm
label_desc: Use image node_id 3 for calling the DistributedGroupedDataParallel obj's underlying member method code with example usage: obj.forward() and returns: self