lora_layers = filter(lambda p: p.requires_grad, unet.parameters())  The optimizer is initialized with the lora_layers because these are the only weights that’ll be optimized:   Copied optimizer = optimizer_cls(