Once gradient tensors are aggregated across all workers, this hook
   applies compression as follows:


Views the input flattened 1D gradient tensor as a square-shaped
      tensor M with 0 paddings;


Creates two low-rank tensors P and Q for decomposing M, such
      that M = PQ^T, where Q is initialized from a standard normal
      distribution and orthogonalized;


Computes P, which is equal to MQ;


Allreduces P;


Orthogonalizes P;


Computes Q, which is approximately equal to M^TP;


Allreduces Q;


Computes M, which is approximately equal to PQ^T.


Truncates the input tensor to the original length.


Note that this communication hook enforces vanilla allreduce for
   the first "state.start_powerSGD_iter" iterations. This not only
   gives the user more control over the tradeoff between speedup and
   accuracy, but also helps abstract away some complexity of the
   internal optimization of DDP for future communication hook
   developers.