class_docstr: A wrapper for the All-Gather function to support auto-differentiation.
class_docstr_tok: ["A","wrapper","for","the","All-Gather","function","to","support","auto-differentiation","."]
code: def forward(ctx, inp, rank, world_size, group):
      tensor_list = [torch.empty_like(inp) for _ in range(world_size)]
      torch.distributed.all_gather(tensor_list, inp, group=group)
      torch.cuda.synchronize()
      output = torch.cat(tensor_list, dim=0)
      ctx.args = rank, inp.shape[0]
      return output
  
imports: import torch
  from torch.autograd import Function
  import fmoe_cuda
  from .utils import get_torch_default_comm
label_desc: Use image node_id 1 for calling the AllGather obj's underlying member method code with example usage: obj.forward(ctx, inp, rank, world_size, group) and returns: output