(Beta) Fully Sharded Data Parallel (FSDP) API
FSDP API helps easily scale large model training by sharding a model’s parameters, gradients and optimizer states across data parallel workers while maintaining the simplicity of data parallelism. The prototype version was released in PyTorch 1.11 with a minimum set of features that helped scaling tests of models with up to 1T parameters. 
In this beta release, FSDP API added the following features to support various production workloads. Highlights of the the newly added features in this beta release include: