model quality will need more training steps to reach or will
         never reach and yield loss in accuracy.
     1.2. The increase of "matrix_approximation_rank" can
     substantially increase the computation costs of the
     compression, and the accuracy may not be further improved
     beyond a certain "matrix_approximation_rank" threshold.

To tune "matrix_approximation_rank", we suggest to start from 1 and
   increase by factors of 2 (like an exponential grid search, 1, 2, 4,
   ...), until a satisfactory accuracy is reached. Typically only a
   small value 1-4 is used. For some NLP tasks (as shown in Appendix D
   of the original paper), this value has been increased to 32.

"start_powerSGD_iter" defers PowerSGD compression until step
      "start_powerSGD_iter", and vanilla allreduce runs prior to step
      "start_powerSGD_iter". This hybrid scheme of **vanilla allreduce
PowerSGD** can effectively improve the accuracy, even a


