xFormers (take a look at the installation instructions if you don’t already have it installed)
set gradients to None
DeepSpeed stage 2 with parameter and optimizer offloading
fp16 mixed precision
DeepSpeed can offload tensors from VRAM to either
CPU or NVME. This requires significantly more RAM (about 25 GB).
You’ll have to configure your environment with accelerate config to enable DeepSpeed stage 2.
The configuration file should look like thi