Controlled generation Controlling outputs generated by diffusion models has been long pursued by the community and is now an active research topic. In many popular diffusion models, subtle changes in inputs, both images and text prompts, can drastically change outputs. In an ideal world we want to be able to control how semantics are preserved and changed. Most examples of preserving semantics reduce to being able to accurately map a change in input to a change in output. I.e. adding an adjective to a subject in a prompt preserves the entire image, only modifying the changed subject. Or, image variation of a particular subject preserves the subject’s pose. Additionally, there are qualities of generated images that we would like to influence beyond semantic preservation. I.e. in general, we would like our outputs to be of good quality, adhere to a particular style, or be realistic. We will document some of the techniques diffusers supports to control generation of diffusion models. Much is cutting edge research and can be quite nuanced. If something needs clarifying or you have a suggestion, don’t hesitate to open a discussion on the forum or a GitHub issue. We provide a high level explanation of how the generation can be controlled as well as a snippet of the technicals. For more in depth explanations on the technicals, the original papers which are linked from the pipelines are always the best resources. Depending on the use case, one should choose a technique accordingly. In many cases, these techniques can be combined. For example, one can combine Textual Inversion with SEGA to provide more semantic guidance to the outputs generated using Textual Inversion. Unless otherwise mentioned, these are techniques that work with existing models and don’t require their own weights. InstructPix2Pix Pix2Pix Zero Attend and Excite Semantic Guidance Self-attention Guidance Depth2Image MultiDiffusion Panorama DreamBooth Textual Inversion ControlNet Prompt Weighting Custom Diffusion Model Editing DiffEdit T2I-Adapter FABRIC For convenience, we provide a table to denote which methods are inference-only and which require fine-tuning/training. Method Inference only Requires training / fine-tuning Comments InstructPix2Pix ✅ ❌ Can additionally befine-tuned for better performance on specific edit instructions. Pix2Pix Zero ✅ ❌  Attend and Excite ✅ ❌  Semantic Guidance ✅ ❌  Self-attention Guidance ✅ ❌  Depth2Image ✅ ❌  MultiDiffusion Panorama ✅ ❌  DreamBooth ❌ ✅  Textual Inversion ❌ ✅  ControlNet ✅ ❌ A ControlNet can be trained/fine-tuned ona custom conditioning. Prompt Weighting ✅ ❌  Custom Diffusion ❌ ✅  Model Editing ✅ ❌  DiffEdit ✅ ❌  T2I-Adapter ✅ ❌  Fabric ✅ ❌   InstructPix2Pix Paper InstructPix2Pix is fine-tuned from Stable Diffusion to support editing input images. It takes as inputs an image and a prompt describing an edit, and it outputs the edited image.