loss.backward()
     optimizer.step()
```
The experiments are conducted on us-central1 GCP cluster. Each machine has 4 NVIDIA Tesla T4 GPUs with 16 GB memory per GPU, connected through a 32 Gbit/s ethernet network. Each instance also features 96 vCPUs, 360 GB RAM.


Architecture
   
ResNet18 (45MB)
   


Workers
   
64, 128, 256
   


Backend
   
NCCL
   


GPU
   
Tesla T4, 16 GB memory
   


Batch size
   
32 x ## of workers
   


Straggler Duration
   
1 sec
   


Straggler Rate
   
1% on 128 and 256 GPUs, 2% on 64 GPUs
   


We used multiple configurations for both local SGD and hierarchical SGD. Local SGD runs global synchronization every 2, 4, and 8 steps, respectively.