Next, we will introduce the background, motivation, our idea, design, and how we implement the algorithm and system with PyTorch Distributed APIs.

Paper: http://proceedings.mlr.press/v139/he21a.html
Source Code: https://DistML.ai.
Slides: https://docs.google.com/presentation/d/1t6HWL33KIQo2as0nSHeBpXYtTBcy0nXCoLiKd0EashY/edit?usp=sharing

Introduction



Figure 1: the Parameter Number of Transformer Models Increases Dramatically.
