```
PyTorch/XLA FSDP is now integrated in HuggingFace trainer class (PR) enabling users to train much larger models on PyTorch/XLA (official Hugging Face documentation). A 16B parameters GPT2 model trained on Cloud TPU v4-64 with this FSDP configuration achieved 39% hardware utilization.


TPU Accelerator - Num Devices

v4-64
   


GPT2 Parameter Count

16B
   


Layers Wrapped with FSDP

GPT2Block
   


TFLOPs / Chip

275
   


PFLOPs / Step

50
   

