class_docstr: Pipeline for text-to-image generation using Stable Diffusion XL with ControlNet-XS guidance.
  
  This model inherits from [`DiffusionPipeline`]. Check the superclass documentation for the generic methods
  implemented for all pipelines (downloading, saving, running on a particular device, etc.).
  
  The pipeline also inherits the following loading methods:
      - [`~loaders.TextualInversionLoaderMixin.load_textual_inversion`] for loading textual inversion embeddings
      - [`~loaders.StableDiffusionXLLoraLoaderMixin.load_lora_weights`] for loading LoRA weights
      - [`~loaders.StableDiffusionXLLoraLoaderMixin.save_lora_weights`] for saving LoRA weights
      - [`~loaders.FromSingleFileMixin.from_single_file`] for loading `.ckpt` files
  
  Args:
      vae ([`AutoencoderKL`]):
          Variational Auto-Encoder (VAE) model to encode and decode images to and from latent representations.
      text_encoder ([`~transformers.CLIPTextModel`]):
          Frozen text-encoder ([clip-vit-large-patch14](https://huggingface.co/openai/clip-vit-large-patch14)).
      text_encoder_2 ([`~transformers.CLIPTextModelWithProjection`]):
          Second frozen text-encoder
          ([laion/CLIP-ViT-bigG-14-laion2B-39B-b160k](https://huggingface.co/laion/CLIP-ViT-bigG-14-laion2B-39B-b160k)).
      tokenizer ([`~transformers.CLIPTokenizer`]):
          A `CLIPTokenizer` to tokenize text.
      tokenizer_2 ([`~transformers.CLIPTokenizer`]):
          A `CLIPTokenizer` to tokenize text.
      unet ([`UNet2DConditionModel`]):
          A `UNet2DConditionModel` to denoise the encoded image latents.
      controlnet ([`ControlNetXSModel`]:
          Provides additional conditioning to the `unet` during the denoising process.
      scheduler ([`SchedulerMixin`]):
          A scheduler to be used in combination with `unet` to denoise the encoded image latents. Can be one of
          [`DDIMScheduler`], [`LMSDiscreteScheduler`], or [`PNDMScheduler`].
      force_zeros_for_empty_prompt (`bool`, *optional*, defaults to `"True"`):
          Whether the negative prompt embeddings should always be set to 0. Also see the config of
          `stabilityai/stable-diffusion-xl-base-1-0`.
      add_watermarker (`bool`, *optional*):
          Whether to use the [invisible_watermark](https://github.com/ShieldMnt/invisible-watermark/) library to
          watermark output images. If not defined, it defaults to `True` if the package is installed; otherwise no
          watermarker is used.
docstr: The call function to the pipeline for generation.
  
  Args:
      prompt (`str` or `List[str]`, *optional*):
          The prompt or prompts to guide image generation. If not defined, you need to pass `prompt_embeds`.
      prompt_2 (`str` or `List[str]`, *optional*):
          The prompt or prompts to be sent to `tokenizer_2` and `text_encoder_2`. If not defined, `prompt` is
          used in both text-encoders.
      image (`torch.FloatTensor`, `PIL.Image.Image`, `np.ndarray`, `List[torch.FloatTensor]`, `List[PIL.Image.Image]`, `List[np.ndarray]`,
              `List[List[torch.FloatTensor]]`, `List[List[np.ndarray]]` or `List[List[PIL.Image.Image]]`):
          The ControlNet input condition to provide guidance to the `unet` for generation. If the type is
          specified as `torch.FloatTensor`, it is passed to ControlNet as is. `PIL.Image.Image` can also be
          accepted as an image. The dimensions of the output image defaults to `image`'s dimensions. If height
          and/or width are passed, `image` is resized accordingly. If multiple ControlNets are specified in
          `init`, images must be passed as a list such that each element of the list can be correctly batched for
          input to a single ControlNet.
      height (`int`, *optional*, defaults to `self.unet.config.sample_size * self.vae_scale_factor`):
          The height in pixels of the generated image. Anything below 512 pixels won't work well for
          [stabilityai/stable-diffusion-xl-base-1.0](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0)
          and checkpoints that are not specifically fine-tuned on low resolutions.
      width (`int`, *optional*, defaults to `self.unet.config.sample_size * self.vae_scale_factor`):
          The width in pixels of the generated image. Anything below 512 pixels won't work well for
          [stabilityai/stable-diffusion-xl-base-1.0](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0)
          and checkpoints that are not specifically fine-tuned on low resolutions.
      num_inference_steps (`int`, *optional*, defaults to 50):
          The number of denoising steps. More denoising steps usually lead to a higher quality image at the
          expense of slower inference.
      guidance_scale (`float`, *optional*, defaults to 5.0):
          A higher guidance scale value encourages the model to generate images closely linked to the text
          `prompt` at the expense of lower image quality. Guidance scale is enabled when `guidance_scale > 1`.
      negative_prompt (`str` or `List[str]`, *optional*):
          The prompt or prompts to guide what to not include in image generation. If not defined, you need to
          pass `negative_prompt_embeds` instead. Ignored when not using guidance (`guidance_scale < 1`).
      negative_prompt_2 (`str` or `List[str]`, *optional*):
          The prompt or prompts to guide what to not include in image generation. This is sent to `tokenizer_2`
          and `text_encoder_2`. If not defined, `negative_prompt` is used in both text-encoders.
      num_images_per_prompt (`int`, *optional*, defaults to 1):
          The number of images to generate per prompt.
      eta (`float`, *optional*, defaults to 0.0):
          Corresponds to parameter eta (η) from the [DDIM](https://arxiv.org/abs/2010.02502) paper. Only applies
          to the [`~schedulers.DDIMScheduler`], and is ignored in other schedulers.
      generator (`torch.Generator` or `List[torch.Generator]`, *optional*):
          A [`torch.Generator`](https://pytorch.org/docs/stable/generated/torch.Generator.html) to make
          generation deterministic.
      latents (`torch.FloatTensor`, *optional*):
          Pre-generated noisy latents sampled from a Gaussian distribution, to be used as inputs for image
          generation. Can be used to tweak the same generation with different prompts. If not provided, a latents
          tensor is generated by sampling using the supplied random `generator`.
      prompt_embeds (`torch.FloatTensor`, *optional*):
          Pre-generated text embeddings. Can be used to easily tweak text inputs (prompt weighting). If not
          provided, text embeddings are generated from the `prompt` input argument.
      negative_prompt_embeds (`torch.FloatTensor`, *optional*):
          Pre-generated negative text embeddings. Can be used to easily tweak text inputs (prompt weighting). If
          not provided, `negative_prompt_embeds` are generated from the `negative_prompt` input argument.
      pooled_prompt_embeds (`torch.FloatTensor`, *optional*):
          Pre-generated pooled text embeddings. Can be used to easily tweak text inputs (prompt weighting). If
          not provided, pooled text embeddings are generated from `prompt` input argument.
      negative_pooled_prompt_embeds (`torch.FloatTensor`, *optional*):
          Pre-generated negative pooled text embeddings. Can be used to easily tweak text inputs (prompt
          weighting). If not provided, pooled `negative_prompt_embeds` are generated from `negative_prompt` input
          argument.
      output_type (`str`, *optional*, defaults to `"pil"`):
          The output format of the generated image. Choose between `PIL.Image` or `np.array`.
      return_dict (`bool`, *optional*, defaults to `True`):
          Whether or not to return a [`~pipelines.stable_diffusion.StableDiffusionPipelineOutput`] instead of a
          plain tuple.
      callback (`Callable`, *optional*):
          A function that calls every `callback_steps` steps during inference. The function is called with the
          following arguments: `callback(step: int, timestep: int, latents: torch.FloatTensor)`.
      callback_steps (`int`, *optional*, defaults to 1):
          The frequency at which the `callback` function is called. If not specified, the callback is called at
          every step.
      cross_attention_kwargs (`dict`, *optional*):
          A kwargs dictionary that if specified is passed along to the [`AttentionProcessor`] as defined in
          [`self.processor`](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/attention_processor.py).
      controlnet_conditioning_scale (`float` or `List[float]`, *optional*, defaults to 1.0):
          The outputs of the ControlNet are multiplied by `controlnet_conditioning_scale` before they are added
          to the residual in the original `unet`.
      control_guidance_start (`float`, *optional*, defaults to 0.0):
          The percentage of total steps at which the ControlNet starts applying.
      control_guidance_end (`float`, *optional*, defaults to 1.0):
          The percentage of total steps at which the ControlNet stops applying.
      original_size (`Tuple[int]`, *optional*, defaults to (1024, 1024)):
          If `original_size` is not the same as `target_size` the image will appear to be down- or upsampled.
          `original_size` defaults to `(width, height)` if not specified. Part of SDXL's micro-conditioning as
          explained in section 2.2 of
          [https://huggingface.co/papers/2307.01952](https://huggingface.co/papers/2307.01952).
      crops_coords_top_left (`Tuple[int]`, *optional*, defaults to (0, 0)):
          `crops_coords_top_left` can be used to generate an image that appears to be "cropped" from the position
          `crops_coords_top_left` downwards. Favorable, well-centered images are usually achieved by setting
          `crops_coords_top_left` to (0, 0). Part of SDXL's micro-conditioning as explained in section 2.2 of
          [https://huggingface.co/papers/2307.01952](https://huggingface.co/papers/2307.01952).
      target_size (`Tuple[int]`, *optional*, defaults to (1024, 1024)):
          For most cases, `target_size` should be set to the desired height and width of the generated image. If
          not specified it will default to `(width, height)`. Part of SDXL's micro-conditioning as explained in
          section 2.2 of [https://huggingface.co/papers/2307.01952](https://huggingface.co/papers/2307.01952).
      negative_original_size (`Tuple[int]`, *optional*, defaults to (1024, 1024)):
          To negatively condition the generation process based on a specific image resolution. Part of SDXL's
          micro-conditioning as explained in section 2.2 of
          [https://huggingface.co/papers/2307.01952](https://huggingface.co/papers/2307.01952). For more
          information, refer to this issue thread: https://github.com/huggingface/diffusers/issues/4208.
      negative_crops_coords_top_left (`Tuple[int]`, *optional*, defaults to (0, 0)):
          To negatively condition the generation process based on a specific crop coordinates. Part of SDXL's
          micro-conditioning as explained in section 2.2 of
          [https://huggingface.co/papers/2307.01952](https://huggingface.co/papers/2307.01952). For more
          information, refer to this issue thread: https://github.com/huggingface/diffusers/issues/4208.
      negative_target_size (`Tuple[int]`, *optional*, defaults to (1024, 1024)):
          To negatively condition the generation process based on a target image resolution. It should be as same
          as the `target_size` for most cases. Part of SDXL's micro-conditioning as explained in section 2.2 of
          [https://huggingface.co/papers/2307.01952](https://huggingface.co/papers/2307.01952). For more
          information, refer to this issue thread: https://github.com/huggingface/diffusers/issues/4208.
      clip_skip (`int`, *optional*):
          Number of layers to be skipped from CLIP while computing the prompt embeddings. A value of 1 means that
          the output of the pre-final layer will be used for computing the prompt embeddings.
  
  Examples:
  
  Returns:
      [`~pipelines.stable_diffusion.StableDiffusionXLPipelineOutput`] or `tuple`:
          If `return_dict` is `True`, [`~pipelines.stable_diffusion.StableDiffusionXLPipelineOutput`] is
          returned, otherwise a `tuple` is returned containing the output images.
code: def __call__(
      self,
      prompt: Union[str, List[str]] = None,
      prompt_2: Optional[Union[str, List[str]]] = None,
      image: PipelineImageInput = None,
      height: Optional[int] = None,
      width: Optional[int] = None,
      num_inference_steps: int = 50,
      guidance_scale: float = 5.0,
      negative_prompt: Optional[Union[str, List[str]]] = None,
      negative_prompt_2: Optional[Union[str, List[str]]] = None,
      num_images_per_prompt: Optional[int] = 1,
      eta: float = 0.0,
      generator: Optional[
          Union[torch.Generator, List[torch.Generator]]
      ] = None,
      latents: Optional[torch.FloatTensor] = None,
      prompt_embeds: Optional[torch.FloatTensor] = None,
      negative_prompt_embeds: Optional[torch.FloatTensor] = None,
      pooled_prompt_embeds: Optional[torch.FloatTensor] = None,
      negative_pooled_prompt_embeds: Optional[torch.FloatTensor] = None,
      output_type: Optional[str] = "pil",
      return_dict: bool = True,
      callback: Optional[
          Callable[[int, int, torch.FloatTensor], None]
      ] = None,
      callback_steps: int = 1,
      cross_attention_kwargs: Optional[Dict[str, Any]] = None,
      controlnet_conditioning_scale: Union[float, List[float]] = 1.0,
      control_guidance_start: float = 0.0,
      control_guidance_end: float = 1.0,
      original_size: Tuple[int, int] = None,
      crops_coords_top_left: Tuple[int, int] = (0, 0),
      target_size: Tuple[int, int] = None,
      negative_original_size: Optional[Tuple[int, int]] = None,
      negative_crops_coords_top_left: Tuple[int, int] = (0, 0),
      negative_target_size: Optional[Tuple[int, int]] = None,
      clip_skip: Optional[int] = None,
  ):
      r"""
      The call function to the pipeline for generation.
  
      Args:
          prompt (`str` or `List[str]`, *optional*):
              The prompt or prompts to guide image generation. If not defined, you need to pass `prompt_embeds`.
          prompt_2 (`str` or `List[str]`, *optional*):
              The prompt or prompts to be sent to `tokenizer_2` and `text_encoder_2`. If not defined, `prompt` is
              used in both text-encoders.
          image (`torch.FloatTensor`, `PIL.Image.Image`, `np.ndarray`, `List[torch.FloatTensor]`, `List[PIL.Image.Image]`, `List[np.ndarray]`,
                  `List[List[torch.FloatTensor]]`, `List[List[np.ndarray]]` or `List[List[PIL.Image.Image]]`):
              The ControlNet input condition to provide guidance to the `unet` for generation. If the type is
              specified as `torch.FloatTensor`, it is passed to ControlNet as is. `PIL.Image.Image` can also be
              accepted as an image. The dimensions of the output image defaults to `image`'s dimensions. If height
              and/or width are passed, `image` is resized accordingly. If multiple ControlNets are specified in
              `init`, images must be passed as a list such that each element of the list can be correctly batched for
              input to a single ControlNet.
          height (`int`, *optional*, defaults to `self.unet.config.sample_size * self.vae_scale_factor`):
              The height in pixels of the generated image. Anything below 512 pixels won't work well for
              [stabilityai/stable-diffusion-xl-base-1.0](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0)
              and checkpoints that are not specifically fine-tuned on low resolutions.
          width (`int`, *optional*, defaults to `self.unet.config.sample_size * self.vae_scale_factor`):
              The width in pixels of the generated image. Anything below 512 pixels won't work well for
              [stabilityai/stable-diffusion-xl-base-1.0](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0)
              and checkpoints that are not specifically fine-tuned on low resolutions.
          num_inference_steps (`int`, *optional*, defaults to 50):
              The number of denoising steps. More denoising steps usually lead to a higher quality image at the
              expense of slower inference.
          guidance_scale (`float`, *optional*, defaults to 5.0):
              A higher guidance scale value encourages the model to generate images closely linked to the text
              `prompt` at the expense of lower image quality. Guidance scale is enabled when `guidance_scale > 1`.
          negative_prompt (`str` or `List[str]`, *optional*):
              The prompt or prompts to guide what to not include in image generation. If not defined, you need to
              pass `negative_prompt_embeds` instead. Ignored when not using guidance (`guidance_scale < 1`).
          negative_prompt_2 (`str` or `List[str]`, *optional*):
              The prompt or prompts to guide what to not include in image generation. This is sent to `tokenizer_2`
              and `text_encoder_2`. If not defined, `negative_prompt` is used in both text-encoders.
          num_images_per_prompt (`int`, *optional*, defaults to 1):
              The number of images to generate per prompt.
          eta (`float`, *optional*, defaults to 0.0):
              Corresponds to parameter eta (η) from the [DDIM](https://arxiv.org/abs/2010.02502) paper. Only applies
              to the [`~schedulers.DDIMScheduler`], and is ignored in other schedulers.
          generator (`torch.Generator` or `List[torch.Generator]`, *optional*):
              A [`torch.Generator`](https://pytorch.org/docs/stable/generated/torch.Generator.html) to make
              generation deterministic.
          latents (`torch.FloatTensor`, *optional*):
              Pre-generated noisy latents sampled from a Gaussian distribution, to be used as inputs for image
              generation. Can be used to tweak the same generation with different prompts. If not provided, a latents
              tensor is generated by sampling using the supplied random `generator`.
          prompt_embeds (`torch.FloatTensor`, *optional*):
              Pre-generated text embeddings. Can be used to easily tweak text inputs (prompt weighting). If not
              provided, text embeddings are generated from the `prompt` input argument.
          negative_prompt_embeds (`torch.FloatTensor`, *optional*):
              Pre-generated negative text embeddings. Can be used to easily tweak text inputs (prompt weighting). If
              not provided, `negative_prompt_embeds` are generated from the `negative_prompt` input argument.
          pooled_prompt_embeds (`torch.FloatTensor`, *optional*):
              Pre-generated pooled text embeddings. Can be used to easily tweak text inputs (prompt weighting). If
              not provided, pooled text embeddings are generated from `prompt` input argument.
          negative_pooled_prompt_embeds (`torch.FloatTensor`, *optional*):
              Pre-generated negative pooled text embeddings. Can be used to easily tweak text inputs (prompt
              weighting). If not provided, pooled `negative_prompt_embeds` are generated from `negative_prompt` input
              argument.
          output_type (`str`, *optional*, defaults to `"pil"`):
              The output format of the generated image. Choose between `PIL.Image` or `np.array`.
          return_dict (`bool`, *optional*, defaults to `True`):
              Whether or not to return a [`~pipelines.stable_diffusion.StableDiffusionPipelineOutput`] instead of a
              plain tuple.
          callback (`Callable`, *optional*):
              A function that calls every `callback_steps` steps during inference. The function is called with the
              following arguments: `callback(step: int, timestep: int, latents: torch.FloatTensor)`.
          callback_steps (`int`, *optional*, defaults to 1):
              The frequency at which the `callback` function is called. If not specified, the callback is called at
              every step.
          cross_attention_kwargs (`dict`, *optional*):
              A kwargs dictionary that if specified is passed along to the [`AttentionProcessor`] as defined in
              [`self.processor`](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/attention_processor.py).
          controlnet_conditioning_scale (`float` or `List[float]`, *optional*, defaults to 1.0):
              The outputs of the ControlNet are multiplied by `controlnet_conditioning_scale` before they are added
              to the residual in the original `unet`.
          control_guidance_start (`float`, *optional*, defaults to 0.0):
              The percentage of total steps at which the ControlNet starts applying.
          control_guidance_end (`float`, *optional*, defaults to 1.0):
              The percentage of total steps at which the ControlNet stops applying.
          original_size (`Tuple[int]`, *optional*, defaults to (1024, 1024)):
              If `original_size` is not the same as `target_size` the image will appear to be down- or upsampled.
              `original_size` defaults to `(width, height)` if not specified. Part of SDXL's micro-conditioning as
              explained in section 2.2 of
              [https://huggingface.co/papers/2307.01952](https://huggingface.co/papers/2307.01952).
          crops_coords_top_left (`Tuple[int]`, *optional*, defaults to (0, 0)):
              `crops_coords_top_left` can be used to generate an image that appears to be "cropped" from the position
              `crops_coords_top_left` downwards. Favorable, well-centered images are usually achieved by setting
              `crops_coords_top_left` to (0, 0). Part of SDXL's micro-conditioning as explained in section 2.2 of
              [https://huggingface.co/papers/2307.01952](https://huggingface.co/papers/2307.01952).
          target_size (`Tuple[int]`, *optional*, defaults to (1024, 1024)):
              For most cases, `target_size` should be set to the desired height and width of the generated image. If
              not specified it will default to `(width, height)`. Part of SDXL's micro-conditioning as explained in
              section 2.2 of [https://huggingface.co/papers/2307.01952](https://huggingface.co/papers/2307.01952).
          negative_original_size (`Tuple[int]`, *optional*, defaults to (1024, 1024)):
              To negatively condition the generation process based on a specific image resolution. Part of SDXL's
              micro-conditioning as explained in section 2.2 of
              [https://huggingface.co/papers/2307.01952](https://huggingface.co/papers/2307.01952). For more
              information, refer to this issue thread: https://github.com/huggingface/diffusers/issues/4208.
          negative_crops_coords_top_left (`Tuple[int]`, *optional*, defaults to (0, 0)):
              To negatively condition the generation process based on a specific crop coordinates. Part of SDXL's
              micro-conditioning as explained in section 2.2 of
              [https://huggingface.co/papers/2307.01952](https://huggingface.co/papers/2307.01952). For more
              information, refer to this issue thread: https://github.com/huggingface/diffusers/issues/4208.
          negative_target_size (`Tuple[int]`, *optional*, defaults to (1024, 1024)):
              To negatively condition the generation process based on a target image resolution. It should be as same
              as the `target_size` for most cases. Part of SDXL's micro-conditioning as explained in section 2.2 of
              [https://huggingface.co/papers/2307.01952](https://huggingface.co/papers/2307.01952). For more
              information, refer to this issue thread: https://github.com/huggingface/diffusers/issues/4208.
          clip_skip (`int`, *optional*):
              Number of layers to be skipped from CLIP while computing the prompt embeddings. A value of 1 means that
              the output of the pre-final layer will be used for computing the prompt embeddings.
  
      Examples:
  
      Returns:
          [`~pipelines.stable_diffusion.StableDiffusionXLPipelineOutput`] or `tuple`:
              If `return_dict` is `True`, [`~pipelines.stable_diffusion.StableDiffusionXLPipelineOutput`] is
              returned, otherwise a `tuple` is returned containing the output images.
      """
      controlnet = (
          self.controlnet._orig_mod
          if is_compiled_module(self.controlnet)
          else self.controlnet
      )
  
      # 1. Check inputs. Raise error if not correct
      self.check_inputs(
          prompt,
          prompt_2,
          image,
          callback_steps,
          negative_prompt,
          negative_prompt_2,
          prompt_embeds,
          negative_prompt_embeds,
          pooled_prompt_embeds,
          negative_pooled_prompt_embeds,
          controlnet_conditioning_scale,
          control_guidance_start,
          control_guidance_end,
      )
  
      # 2. Define call parameters
      if prompt is not None and isinstance(prompt, str):
          batch_size = 1
      elif prompt is not None and isinstance(prompt, list):
          batch_size = len(prompt)
      else:
          batch_size = prompt_embeds.shape[0]
  
      device = self._execution_device
      # here `guidance_scale` is defined analog to the guidance weight `w` of equation (2)
      # of the Imagen paper: https://arxiv.org/pdf/2205.11487.pdf . `guidance_scale = 1`
      # corresponds to doing no classifier free guidance.
      do_classifier_free_guidance = guidance_scale > 1.0
  
      # 3. Encode input prompt
      text_encoder_lora_scale = (
          cross_attention_kwargs.get("scale", None)
          if cross_attention_kwargs is not None
          else None
      )
      (
          prompt_embeds,
          negative_prompt_embeds,
          pooled_prompt_embeds,
          negative_pooled_prompt_embeds,
      ) = self.encode_prompt(
          prompt,
          prompt_2,
          device,
          num_images_per_prompt,
          do_classifier_free_guidance,
          negative_prompt,
          negative_prompt_2,
          prompt_embeds=prompt_embeds,
          negative_prompt_embeds=negative_prompt_embeds,
          pooled_prompt_embeds=pooled_prompt_embeds,
          negative_pooled_prompt_embeds=negative_pooled_prompt_embeds,
          lora_scale=text_encoder_lora_scale,
          clip_skip=clip_skip,
      )
  
      # 4. Prepare image
      if isinstance(controlnet, ControlNetXSModel):
          image = self.prepare_image(
              image=image,
              width=width,
              height=height,
              batch_size=batch_size * num_images_per_prompt,
              num_images_per_prompt=num_images_per_prompt,
              device=device,
              dtype=controlnet.dtype,
              do_classifier_free_guidance=do_classifier_free_guidance,
          )
          height, width = image.shape[-2:]
      else:
          assert False
  
      # 5. Prepare timesteps
      self.scheduler.set_timesteps(num_inference_steps, device=device)
      timesteps = self.scheduler.timesteps
  
      # 6. Prepare latent variables
      num_channels_latents = self.unet.config.in_channels
      latents = self.prepare_latents(
          batch_size * num_images_per_prompt,
          num_channels_latents,
          height,
          width,
          prompt_embeds.dtype,
          device,
          generator,
          latents,
      )
  
      # 7. Prepare extra step kwargs. TODO: Logic should ideally just be moved out of the pipeline
      extra_step_kwargs = self.prepare_extra_step_kwargs(generator, eta)
  
      # 7.1 Prepare added time ids & embeddings
      if isinstance(image, list):
          original_size = original_size or image[0].shape[-2:]
      else:
          original_size = original_size or image.shape[-2:]
      target_size = target_size or (height, width)
  
      add_text_embeds = pooled_prompt_embeds
      if self.text_encoder_2 is None:
          text_encoder_projection_dim = int(
              pooled_prompt_embeds.shape[-1]
          )
      else:
          text_encoder_projection_dim = (
              self.text_encoder_2.config.projection_dim
          )
  
      add_time_ids = self._get_add_time_ids(
          original_size,
          crops_coords_top_left,
          target_size,
          dtype=prompt_embeds.dtype,
          text_encoder_projection_dim=text_encoder_projection_dim,
      )
  
      if (
          negative_original_size is not None
          and negative_target_size is not None
      ):
          negative_add_time_ids = self._get_add_time_ids(
              negative_original_size,
              negative_crops_coords_top_left,
              negative_target_size,
              dtype=prompt_embeds.dtype,
              text_encoder_projection_dim=text_encoder_projection_dim,
          )
      else:
          negative_add_time_ids = add_time_ids
  
      if do_classifier_free_guidance:
          prompt_embeds = torch.cat(
              [negative_prompt_embeds, prompt_embeds], dim=0
          )
          add_text_embeds = torch.cat(
              [negative_pooled_prompt_embeds, add_text_embeds], dim=0
          )
          add_time_ids = torch.cat(
              [negative_add_time_ids, add_time_ids], dim=0
          )
  
      prompt_embeds = prompt_embeds.to(device)
      add_text_embeds = add_text_embeds.to(device)
      add_time_ids = add_time_ids.to(device).repeat(
          batch_size * num_images_per_prompt, 1
      )
  
      # 8. Denoising loop
      num_warmup_steps = (
          len(timesteps) - num_inference_steps * self.scheduler.order
      )
      is_unet_compiled = is_compiled_module(self.unet)
      is_controlnet_compiled = is_compiled_module(self.controlnet)
      is_torch_higher_equal_2_1 = is_torch_version(">=", "2.1")
      with self.progress_bar(total=num_inference_steps) as progress_bar:
          for i, t in enumerate(timesteps):
              # Relevant thread:
              # https://dev-discuss.pytorch.org/t/cudagraphs-in-pytorch-2-0/1428
              if (
                  is_unet_compiled and is_controlnet_compiled
              ) and is_torch_higher_equal_2_1:
                  torch._inductor.cudagraph_mark_step_begin()
              # expand the latents if we are doing classifier free guidance
              latent_model_input = (
                  torch.cat([latents] * 2)
                  if do_classifier_free_guidance
                  else latents
              )
              latent_model_input = self.scheduler.scale_model_input(
                  latent_model_input, t
              )
  
              added_cond_kwargs = {
                  "text_embeds": add_text_embeds,
                  "time_ids": add_time_ids,
              }
  
              # predict the noise residual
              dont_control = (
                  i / len(timesteps) < control_guidance_start
                  or (i + 1) / len(timesteps) > control_guidance_end
              )
              if dont_control:
                  noise_pred = self.unet(
                      sample=latent_model_input,
                      timestep=t,
                      encoder_hidden_states=prompt_embeds,
                      cross_attention_kwargs=cross_attention_kwargs,
                      added_cond_kwargs=added_cond_kwargs,
                      return_dict=True,
                  ).sample
              else:
                  noise_pred = self.controlnet(
                      base_model=self.unet,
                      sample=latent_model_input,
                      timestep=t,
                      encoder_hidden_states=prompt_embeds,
                      controlnet_cond=image,
                      conditioning_scale=controlnet_conditioning_scale,
                      cross_attention_kwargs=cross_attention_kwargs,
                      added_cond_kwargs=added_cond_kwargs,
                      return_dict=True,
                  ).sample
  
              # perform guidance
              if do_classifier_free_guidance:
                  noise_pred_uncond, noise_pred_text = noise_pred.chunk(
                      2
                  )
                  noise_pred = noise_pred_uncond + guidance_scale * (
                      noise_pred_text - noise_pred_uncond
                  )
  
              # compute the previous noisy sample x_t -> x_t-1
              latents = self.scheduler.step(
                  noise_pred,
                  t,
                  latents,
                  **extra_step_kwargs,
                  return_dict=False,
              )[0]
  
              # call the callback, if provided
              if i == len(timesteps) - 1 or (
                  (i + 1) > num_warmup_steps
                  and (i + 1) % self.scheduler.order == 0
              ):
                  progress_bar.update()
                  if callback is not None and i % callback_steps == 0:
                      step_idx = i // getattr(
                          self.scheduler, "order", 1
                      )
                      callback(step_idx, t, latents)
  
      # manually for max memory savings
      if (
          self.vae.dtype == torch.float16
          and self.vae.config.force_upcast
      ):
          self.upcast_vae()
          latents = latents.to(
              next(iter(self.vae.post_quant_conv.parameters())).dtype
          )
  
      if not output_type == "latent":
          # make sure the VAE is in float32 mode, as it overflows in float16
          needs_upcasting = (
              self.vae.dtype == torch.float16
              and self.vae.config.force_upcast
          )
  
          if needs_upcasting:
              self.upcast_vae()
              latents = latents.to(
                  next(
                      iter(self.vae.post_quant_conv.parameters())
                  ).dtype
              )
  
          image = self.vae.decode(
              latents / self.vae.config.scaling_factor,
              return_dict=False,
          )[0]
  
          # cast back to fp16 if needed
          if needs_upcasting:
              self.vae.to(dtype=torch.float16)
      else:
          image = latents
  
      if not output_type == "latent":
          # apply watermark if available
          if self.watermark is not None:
              image = self.watermark.apply_watermark(image)
  
          image = self.image_processor.postprocess(
              image, output_type=output_type
          )
  
      # Offload all models
      self.maybe_free_model_hooks()
  
      if not return_dict:
          return (image,)
  
      return StableDiffusionXLPipelineOutput(images=image)
  
imports: import inspect
  from typing import Any, Callable, Dict, List, Optional, Tuple, Union
  import numpy
  import PIL.Image
  import torch
  import torch.nn.functional
  from transformers import CLIPTextModel, CLIPTextModelWithProjection, CLIPTokenizer
  from diffusers.image_processor import PipelineImageInput, VaeImageProcessor
  from diffusers.loaders import FromSingleFileMixin, StableDiffusionXLLoraLoaderMixin, TextualInversionLoaderMixin
  from diffusers.models import AutoencoderKL, ControlNetXSModel, UNet2DConditionModel
  from diffusers.models.attention_processor import AttnProcessor2_0, LoRAAttnProcessor2_0, LoRAXFormersAttnProcessor, XFormersAttnProcessor
  from diffusers.models.lora import adjust_lora_scale_text_encoder
  from diffusers.pipelines.pipeline_utils import DiffusionPipeline
  from diffusers.pipelines.stable_diffusion_xl.pipeline_output import StableDiffusionXLPipelineOutput
  from diffusers.schedulers import KarrasDiffusionSchedulers
  from diffusers.utils import USE_PEFT_BACKEND, logging, scale_lora_layers, unscale_lora_layers
  from diffusers.utils.import_utils import is_invisible_watermark_available
  from diffusers.utils.torch_utils import is_compiled_module, is_torch_version, randn_tensor
label_desc: Use image node_id 16 for calling the StableDiffusionXLControlNetXSPipeline obj's underlying member method code with example usage: obj.__call__(prompt, prompt_2, image, height, width, num_inference_steps, guidance_scale, negative_prompt, negative_prompt_2, num_images_per_prompt, eta, generator, latents, prompt_embeds, negative_prompt_embeds, pooled_prompt_embeds, negative_pooled_prompt_embeds, output_type, return_dict, callback, callback_steps, cross_attention_kwargs, controlnet_conditioning_scale, control_guidance_start, control_guidance_end, original_size, crops_coords_top_left, target_size, negative_original_size, negative_crops_coords_top_left, negative_target_size, clip_skip) and returns: StableDiffusionXLPipelineOutput, image