
Since commonly used PyTorch functions are already annotated, their names are automatically shown on the trace. With them, we can roughly divide the trace into the four phases in a training iteration: (1) data loading, (2) forward pass, (3) backward pass, (4) gradient optimization (note: In Figure 5, the “optimizer” phase is from the previous batch while the other three phases are from the current batch).
2.2 Optimizations
We performed four simple optimizations that target the bottlenecks identified above, each requiring only a change in a config parameter or at most a few source lines. They are listed in Figure 6.



Optimization
Amount of changes
Bottlenecks addressed




Tune num_worker_threads by trying a few possible values within the number of CPU cores on each host.
1 source line
GPU totally idle time


Double the batch sizes
2 config parameters
GPU memory under-utilization


