


ddp_model.register_comm_hook(state, powerSGD_hook)



torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook.batched_powerSGD_hook(state, bucket)
This DDP communication hook implements a simplified PowerSGD
   gradient compression algorithm described in the paper. This variant
   does not compress the gradients layer by layer, but instead
   compresses the flattened input tensor that batches all the
   gradients. Therefore, it is faster than "powerSGD_hook()", but
   usually results in a much lower accuracy, unless
   "matrix_approximation_rank" is 1.
Warning:
 Increasing "matrix_approximation_rank" here may not necessarily
 increase the accuracy, because batching per-parameter tensors
 without column/row alignment can destroy low-rank structure.
 Therefore, the user should always consider "powerSGD_hook()"
 first, and only consider this variant when a satisfactory
 accuracy can be achieved when "matrix_approximation_rank" is 1.
