Default Communication Hooks
Default communication hooks are simple stateless hooks, so the
input state in "register_comm_hook" is either a process group or
"None". The input "bucket" is a "torch.distributed.GradBucket" object.
torch.distributed.algorithms.ddp_comm_hooks.default_hooks.allreduce_hook(process_group, bucket)
This DDP communication hook just calls "allreduce" using
   "GradBucket" tensors. Once gradient tensors are aggregated across
   all workers, its "then" callback takes the mean and returns the
   result. If user registers this hook, DDP results is expected to be
   same as the case where no hook was registered. Hence, this won't
   change behavior of DDP and user can use this as a reference or
   modify this hook to log useful information or any other purposes
   while unaffecting DDP behavior.
Example::
      >>> ddp_model.register_comm_hook(process_group, allreduce_hook)
Return type:
      Future[Tensor]