
For some cases, hierarchical SGD could lead to a slightly lower quality than the original model for the same number of training steps (i.e., lower convergence rate), but with a speedup like 2X+ per training step, it is still possible to achieve model parity with more steps but still less total training time.

 
 
 
Limitations
Before applying hierarchical SGD to straggler mitigation, the user should be aware of a few limitations of this approach: