
5. Concluding Remarks
We have demonstrated the use of PyTorch with FairScale’s FullyShardedDataParallel (FSDP) API in writing large vision transformer models. We discuss our techniques for scaling and optimizing these models on a GPU cluster.  We hope that this article can motivate others to develop large-scale ML models with PyTorch and its ecosystem.
References
[1] Masked Autoencoders Are Scalable Vision Learners
[2] Revisiting Weakly Supervised Pre-Training of Visual Perception Models
[3] An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
[4] fairscale.nn.FullyShardedDataParallel
[5] Pipeline parallelism in PyTorch
[6] Automatic Mixed Precision (AMP) in PyTorch