


Figure 9. Speedup Breakdown (ViT on ImageNet)

To understand the efficacy of all four components and their impacts on training speed, we experimented with different combinations and used their training sample throughput (samples/second) and speedup ratio as metrics. Results are illustrated in Figure 9. Key takeaways from these experimental results are:

the main speedup is the result of elastic pipelining which is achieved through the joint use of AutoPipe and AutoDP;
AutoCache's contribution is amplified by AutoDP;
freeze training alone without system-wise adjustment even downgrades the training speed.

Tuning  in Freezing Algorithm


