    )

unet.set_attn_processor(lora_attn_procs)
lora_layers = AttnProcsLayers(unet.attn_processors) The optimizer is initialized with the lora_layers because these are the only weights that’ll be optimized:   Copied optimizer = optimizer_cls(
    lora_layers.parameters(),
    lr=args.learning_rate,
    betas=(args.adam_beta1, args.adam_beta2),
    weight_decay=args.adam_weight_decay,
    eps=args.adam_epsilon,
) Aside from setting up the LoRA l