In this first post, we will provide a quick overview of FSDP and how it can make training large- scale AI models more efficient.  We will highlight briefly the multiple performance options available, and dive deeper into the details on these in upcoming posts.  We will then conclude with an overview on how to leverage AWS parallel cluster for large- scale training with FSDP. 


Optimization 

T5 Model 

Throughput Improvement 



Mixed Precision
   
3 B
   
5x
   


11 B
   
10x
   


Activation Checkpointing (AC)
   
3 B
   
10x
   


11 B
   
100x
   


Transformer Wrapping Policy
   
3 B
   
2x
   
