You can reduce your memory footprint even more by enabling memory efficient attention with xFormers. Make sure you have xFormers installed and pass the --enable_xformers_memory_efficient_attention flag to the training script.