ee the DeepSpeed documentation for more configuration options.
You should also change the default Adam optimizer to DeepSpeed’s optimized version of Adam
deepspeed.ops.adam.DeepSpeedCPUAdam for a substantial speedup. Enabling DeepSpeedCPUAdam requires your system’s CUDA toolchain version to be the same as the one installed with PyTorch. 
8-bit optimizers don’t seem to be compatible with DeepSpeed at the moment.
Launch training with the followin