
layout: blog_detail
title: 'PipeTransformer: Automated Elastic Pipelining for Distributed Training of Large-scale Models'
author: Chaoyang He, Shen Li, Mahdi Soltanolkotabi, and Salman Avestimehr
featured-img: 'assets/images/pipetransformer_overview.png'

In this blog post, we describe the first peer-reviewed research paper that explores accelerating the hybrid of PyTorch DDP (torch.nn.parallel.DistributedDataParallel) [1] and Pipeline (torch.distributed.pipeline) - PipeTransformer: Automated Elastic Pipelining for Distributed Training of Large-scale Models (Transformers such as BERT [2]  and ViT [3]), published at ICML 2021.