
The first screenshot shows that a process has a very high allreduce cost in both the first and the third steps, because this process reaches the synchronization phase earlier than the straggler(s), and it spends more time on waiting. On the other hand, the allreduce cost is relatively small in the second step, this suggests that 1) there is no straggler at this step; or 2) this process is the straggler among all the processes, so it does not need to wait for any other process.

 
Both the 1st and the 3rd Steps Are Slowed Down by Stragglers

The second screenshot shows a normal case without stragglers. In this case, all the gradient synchronizations are relatively short.

