3 B
   
2x
   



11 B
   
Unable to run the experiment without the Transformer wrapping policy.



Full Shard Strategy
   
3 B
   
1.5x
   


11 B
   
Not able to run with Zero2



Performance optimization gains on T5 models over non-optimized.