e to use StableDiffusionXLControlNetPipeline and pass image and controlnet_conditioning_scale.   Copied base = StableDiffusionXLControlNetPipeline(...)
image = base(
    prompt=prompt,
    controlnet_conditioning_scale=0.5,
    image=canny_image,
    num_inference_steps=40,
    denoising_end=0.8,
    output_type="latent",
).images
# rest exactly as with StableDiffusionXLPipeline  MultiControlNet Replace the SDXL model with a model like runwayml