minibatch into multiple microbatches and pipelines the execution of
these microbatches across multiple GPUs. This is outlined in the
figure below:
[image]The figure represents a model with 4 layers placed on 4
   different GPUs (vertical axis). The horizontal axis represents
   training this model through time demonstrating that the GPUs are
   utilized much more efficiently. However, there still exists a
   bubble (as demonstrated in the figure) where certain GPUs are not
   utilized. (image source).
Pipe APIs in PyTorch
class torch.distributed.pipeline.sync.Pipe(module, chunks=1, checkpoint='except_last', deferred_batch_norm=False)
Wraps an arbitrary "nn.Sequential" module to train on using
   synchronous pipeline parallelism. If the module requires lots of
   memory and doesn't fit on a single GPU, pipeline parallelism is a
   useful technique to employ for training.
The implementation is based on the torchgpipe paper.