
Additionally, such a technique can also speed up training by shrinking the size of pipeline bubbles. To explain bubble sizes in a pipeline, Figure 7 depicts how 4 micro-batches run through a 4-device pipeline . In general, the total bubble size is  times per micro-batch forward and backward cost. Therefore, it is clear that shorter pipelines have smaller bubble sizes.
Dynamic Number of Micro-Batches