Pipeline Parallelism
Pipeline parallelism was original introduced in the Gpipe  paper and
is an efficient technique to train large models on multiple GPUs.
Warning:
Pipeline Parallelism is experimental and subject to change.
Model Parallelism using multiple GPUs
Typically for large models which don't fit on a single GPU, model
parallelism is employed where certain parts of the model are placed on
different GPUs. Although, if this is done naively for sequential
models, the training process suffers from GPU under utilization since
only one GPU is active at one time as shown in the figure below:
[image]The figure represents a model with 4 layers placed on 4
   different GPUs (vertical axis). The horizontal axis represents
   training this model through time demonstrating that only 1 GPU is
   utilized at a time (image source).
Pipelined Execution
To alleviate this problem, pipeline parallelism splits the input