on,
answer yes to “Do you want to use DeepSpeed?“. Combining DeepSpeed stage 2, fp16
mixed precision, and offloading both the model parameters and the optimizer state to CPU, it’s
possible to train on under 8 GB VRAM. The drawback is that this requires more system RAM (about 25 GB). See the DeepSpeed documentation for more configuration options.
Changing the default Adam optimizer to DeepSpeed’s special version of Adam
deepspeed.ops.adam.DeepSp