ControlNet ControlNet models are adapters trained on top of another pretrained model. It allows for a greater degree of control over image generation by conditioning the model with an additional input image. The input image can be a canny edge, depth map, human pose, and many more. If you’re training on a GPU with limited vRAM, you should try enabling the gradient_checkpointing, gradient_accumulation_steps, and mixed_precision parameters in the training command. You can also reduce your memory footprint by using memory-efficient attention with xFormers. JAX/Flax training is also supported for efficient training on TPUs and GPUs, but it doesn’t support gradient checkpointing or xFormers. You should have a GPU with >30GB of memory if you want to train faster with Flax. This guide will explore the train_controlnet.py training script to help you become familiar with it, and how you can adapt it for your own use-case. Before running the script, make sure you install the library from source:   Copied git clone https://github.com/huggingface/diffusers