
Figure 7. Pipeline Bubble: , and  denote forward, backward, and the optimizer update of micro-batch  on device , respectively. The total bubble size in each iteration is  times per micro-batch forward and backward cost.
