DDP Communication Hooks
DDP communication hook is a generic interface to control how to
communicate gradients across workers by overriding the vanilla
allreduce in DistributedDataParallel. A few built-in communication
hooks are provided, and users can easily apply any of these hooks to
optimize communication. Besides, the hook interface can also support
user-defined communication strategies for more advanced use cases.
How to Use a Communication Hook?
To use a communication hook, the user just needs to let the DDP model
register the hook before the training loop as below.
"torch.nn.parallel.DistributedDataParallel.register_comm_hook()"
What Does a Communication Hook Operate On?
A communication hook provides a flexible way to allreduce gradients.
Therefore, it mainly operates on the gradients on each replica before
allreduce, which are bucketized to increase the overlap between