
All the groups at the same level have the same number of processes, and the processes in these groups synchronize at the same frequency concurrently, where the synchronization period is pre-defined by the user.
The higher level a group is, the larger synchronization period is used, as the synchronization becomes more expensive.
When multiple overlapping groups are supposed to synchronize according to their periods, to reduce redundant synchronization and avoid data race across groups, only the highest-level group runs synchronization.

The following figure illustrates an example of 4-level hierarchy SGD among 16 processes on 8 machines, each of which has 2 GPUs:

Level 1: Each process runs mini-batch SGD locally;
Level 2: Each 4-process group across 2 machines runs synchronization every 2 steps;
Level 3: Each 8-process group across 4 machines runs synchronization every 4 steps;
