class_docstr: Pipeline for text-to-image generation using Stable Diffusion XL with ControlNet-XS guidance.
  
  This model inherits from [`DiffusionPipeline`]. Check the superclass documentation for the generic methods
  implemented for all pipelines (downloading, saving, running on a particular device, etc.).
  
  The pipeline also inherits the following loading methods:
      - [`~loaders.TextualInversionLoaderMixin.load_textual_inversion`] for loading textual inversion embeddings
      - [`~loaders.StableDiffusionXLLoraLoaderMixin.load_lora_weights`] for loading LoRA weights
      - [`~loaders.StableDiffusionXLLoraLoaderMixin.save_lora_weights`] for saving LoRA weights
      - [`~loaders.FromSingleFileMixin.from_single_file`] for loading `.ckpt` files
  
  Args:
      vae ([`AutoencoderKL`]):
          Variational Auto-Encoder (VAE) model to encode and decode images to and from latent representations.
      text_encoder ([`~transformers.CLIPTextModel`]):
          Frozen text-encoder ([clip-vit-large-patch14](https://huggingface.co/openai/clip-vit-large-patch14)).
      text_encoder_2 ([`~transformers.CLIPTextModelWithProjection`]):
          Second frozen text-encoder
          ([laion/CLIP-ViT-bigG-14-laion2B-39B-b160k](https://huggingface.co/laion/CLIP-ViT-bigG-14-laion2B-39B-b160k)).
      tokenizer ([`~transformers.CLIPTokenizer`]):
          A `CLIPTokenizer` to tokenize text.
      tokenizer_2 ([`~transformers.CLIPTokenizer`]):
          A `CLIPTokenizer` to tokenize text.
      unet ([`UNet2DConditionModel`]):
          A `UNet2DConditionModel` to denoise the encoded image latents.
      controlnet ([`ControlNetXSModel`]:
          Provides additional conditioning to the `unet` during the denoising process.
      scheduler ([`SchedulerMixin`]):
          A scheduler to be used in combination with `unet` to denoise the encoded image latents. Can be one of
          [`DDIMScheduler`], [`LMSDiscreteScheduler`], or [`PNDMScheduler`].
      force_zeros_for_empty_prompt (`bool`, *optional*, defaults to `"True"`):
          Whether the negative prompt embeddings should always be set to 0. Also see the config of
          `stabilityai/stable-diffusion-xl-base-1-0`.
      add_watermarker (`bool`, *optional*):
          Whether to use the [invisible_watermark](https://github.com/ShieldMnt/invisible-watermark/) library to
          watermark output images. If not defined, it defaults to `True` if the package is installed; otherwise no
          watermarker is used.
docstr: Encodes the prompt into text encoder hidden states.
  
  Args:
      prompt (`str` or `List[str]`, *optional*):
          prompt to be encoded
      prompt_2 (`str` or `List[str]`, *optional*):
          The prompt or prompts to be sent to the `tokenizer_2` and `text_encoder_2`. If not defined, `prompt` is
          used in both text-encoders
      device: (`torch.device`):
          torch device
      num_images_per_prompt (`int`):
          number of images that should be generated per prompt
      do_classifier_free_guidance (`bool`):
          whether to use classifier free guidance or not
      negative_prompt (`str` or `List[str]`, *optional*):
          The prompt or prompts not to guide the image generation. If not defined, one has to pass
          `negative_prompt_embeds` instead. Ignored when not using guidance (i.e., ignored if `guidance_scale` is
          less than `1`).
      negative_prompt_2 (`str` or `List[str]`, *optional*):
          The prompt or prompts not to guide the image generation to be sent to `tokenizer_2` and
          `text_encoder_2`. If not defined, `negative_prompt` is used in both text-encoders
      prompt_embeds (`torch.FloatTensor`, *optional*):
          Pre-generated text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not
          provided, text embeddings will be generated from `prompt` input argument.
      negative_prompt_embeds (`torch.FloatTensor`, *optional*):
          Pre-generated negative text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt
          weighting. If not provided, negative_prompt_embeds will be generated from `negative_prompt` input
          argument.
      pooled_prompt_embeds (`torch.FloatTensor`, *optional*):
          Pre-generated pooled text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting.
          If not provided, pooled text embeddings will be generated from `prompt` input argument.
      negative_pooled_prompt_embeds (`torch.FloatTensor`, *optional*):
          Pre-generated negative pooled text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt
          weighting. If not provided, pooled negative_prompt_embeds will be generated from `negative_prompt`
          input argument.
      lora_scale (`float`, *optional*):
          A lora scale that will be applied to all LoRA layers of the text encoder if LoRA layers are loaded.
      clip_skip (`int`, *optional*):
          Number of layers to be skipped from CLIP while computing the prompt embeddings. A value of 1 means that
          the output of the pre-final layer will be used for computing the prompt embeddings.
code: def encode_prompt(
      self,
      prompt: str,
      prompt_2: Optional[str] = None,
      device: Optional[torch.device] = None,
      num_images_per_prompt: int = 1,
      do_classifier_free_guidance: bool = True,
      negative_prompt: Optional[str] = None,
      negative_prompt_2: Optional[str] = None,
      prompt_embeds: Optional[torch.FloatTensor] = None,
      negative_prompt_embeds: Optional[torch.FloatTensor] = None,
      pooled_prompt_embeds: Optional[torch.FloatTensor] = None,
      negative_pooled_prompt_embeds: Optional[torch.FloatTensor] = None,
      lora_scale: Optional[float] = None,
      clip_skip: Optional[int] = None,
  ):
      r"""
      Encodes the prompt into text encoder hidden states.
  
      Args:
          prompt (`str` or `List[str]`, *optional*):
              prompt to be encoded
          prompt_2 (`str` or `List[str]`, *optional*):
              The prompt or prompts to be sent to the `tokenizer_2` and `text_encoder_2`. If not defined, `prompt` is
              used in both text-encoders
          device: (`torch.device`):
              torch device
          num_images_per_prompt (`int`):
              number of images that should be generated per prompt
          do_classifier_free_guidance (`bool`):
              whether to use classifier free guidance or not
          negative_prompt (`str` or `List[str]`, *optional*):
              The prompt or prompts not to guide the image generation. If not defined, one has to pass
              `negative_prompt_embeds` instead. Ignored when not using guidance (i.e., ignored if `guidance_scale` is
              less than `1`).
          negative_prompt_2 (`str` or `List[str]`, *optional*):
              The prompt or prompts not to guide the image generation to be sent to `tokenizer_2` and
              `text_encoder_2`. If not defined, `negative_prompt` is used in both text-encoders
          prompt_embeds (`torch.FloatTensor`, *optional*):
              Pre-generated text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not
              provided, text embeddings will be generated from `prompt` input argument.
          negative_prompt_embeds (`torch.FloatTensor`, *optional*):
              Pre-generated negative text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt
              weighting. If not provided, negative_prompt_embeds will be generated from `negative_prompt` input
              argument.
          pooled_prompt_embeds (`torch.FloatTensor`, *optional*):
              Pre-generated pooled text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting.
              If not provided, pooled text embeddings will be generated from `prompt` input argument.
          negative_pooled_prompt_embeds (`torch.FloatTensor`, *optional*):
              Pre-generated negative pooled text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt
              weighting. If not provided, pooled negative_prompt_embeds will be generated from `negative_prompt`
              input argument.
          lora_scale (`float`, *optional*):
              A lora scale that will be applied to all LoRA layers of the text encoder if LoRA layers are loaded.
          clip_skip (`int`, *optional*):
              Number of layers to be skipped from CLIP while computing the prompt embeddings. A value of 1 means that
              the output of the pre-final layer will be used for computing the prompt embeddings.
      """
      device = device or self._execution_device
  
      # set lora scale so that monkey patched LoRA
      # function of text encoder can correctly access it
      if lora_scale is not None and isinstance(
          self, StableDiffusionXLLoraLoaderMixin
      ):
          self._lora_scale = lora_scale
  
          # dynamically adjust the LoRA scale
          if self.text_encoder is not None:
              if not USE_PEFT_BACKEND:
                  adjust_lora_scale_text_encoder(
                      self.text_encoder, lora_scale
                  )
              else:
                  scale_lora_layers(self.text_encoder, lora_scale)
  
          if self.text_encoder_2 is not None:
              if not USE_PEFT_BACKEND:
                  adjust_lora_scale_text_encoder(
                      self.text_encoder_2, lora_scale
                  )
              else:
                  scale_lora_layers(self.text_encoder_2, lora_scale)
  
      prompt = [prompt] if isinstance(prompt, str) else prompt
  
      if prompt is not None:
          batch_size = len(prompt)
      else:
          batch_size = prompt_embeds.shape[0]
  
      # Define tokenizers and text encoders
      tokenizers = (
          [self.tokenizer, self.tokenizer_2]
          if self.tokenizer is not None
          else [self.tokenizer_2]
      )
      text_encoders = (
          [self.text_encoder, self.text_encoder_2]
          if self.text_encoder is not None
          else [self.text_encoder_2]
      )
  
      if prompt_embeds is None:
          prompt_2 = prompt_2 or prompt
          prompt_2 = (
              [prompt_2] if isinstance(prompt_2, str) else prompt_2
          )
  
          # textual inversion: procecss multi-vector tokens if necessary
          prompt_embeds_list = []
          prompts = [prompt, prompt_2]
          for prompt, tokenizer, text_encoder in zip(
              prompts, tokenizers, text_encoders
          ):
              if isinstance(self, TextualInversionLoaderMixin):
                  prompt = self.maybe_convert_prompt(prompt, tokenizer)
  
              text_inputs = tokenizer(
                  prompt,
                  padding="max_length",
                  max_length=tokenizer.model_max_length,
                  truncation=True,
                  return_tensors="pt",
              )
  
              text_input_ids = text_inputs.input_ids
              untruncated_ids = tokenizer(
                  prompt, padding="longest", return_tensors="pt"
              ).input_ids
  
              if untruncated_ids.shape[-1] >= text_input_ids.shape[
                  -1
              ] and not torch.equal(text_input_ids, untruncated_ids):
                  removed_text = tokenizer.batch_decode(
                      untruncated_ids[
                          :, tokenizer.model_max_length - 1 : -1
                      ]
                  )
                  logger.warning(
                      "The following part of your input was truncated because CLIP can only handle sequences up to"
                      f" {tokenizer.model_max_length} tokens: {removed_text}"
                  )
  
              prompt_embeds = text_encoder(
                  text_input_ids.to(device), output_hidden_states=True
              )
  
              # We are only ALWAYS interested in the pooled output of the final text encoder
              pooled_prompt_embeds = prompt_embeds[0]
              if clip_skip is None:
                  prompt_embeds = prompt_embeds.hidden_states[-2]
              else:
                  # "2" because SDXL always indexes from the penultimate layer.
                  prompt_embeds = prompt_embeds.hidden_states[
                      -(clip_skip + 2)
                  ]
  
              prompt_embeds_list.append(prompt_embeds)
  
          prompt_embeds = torch.concat(prompt_embeds_list, dim=-1)
  
      # get unconditional embeddings for classifier free guidance
      zero_out_negative_prompt = (
          negative_prompt is None
          and self.config.force_zeros_for_empty_prompt
      )
      if (
          do_classifier_free_guidance
          and negative_prompt_embeds is None
          and zero_out_negative_prompt
      ):
          negative_prompt_embeds = torch.zeros_like(prompt_embeds)
          negative_pooled_prompt_embeds = torch.zeros_like(
              pooled_prompt_embeds
          )
      elif (
          do_classifier_free_guidance and negative_prompt_embeds is None
      ):
          negative_prompt = negative_prompt or ""
          negative_prompt_2 = negative_prompt_2 or negative_prompt
  
          # normalize str to list
          negative_prompt = (
              batch_size * [negative_prompt]
              if isinstance(negative_prompt, str)
              else negative_prompt
          )
          negative_prompt_2 = (
              batch_size * [negative_prompt_2]
              if isinstance(negative_prompt_2, str)
              else negative_prompt_2
          )
  
          uncond_tokens: List[str]
          if prompt is not None and type(prompt) is not type(
              negative_prompt
          ):
              raise TypeError(
                  f"`negative_prompt` should be the same type to `prompt`, but got {type(negative_prompt)} !="
                  f" {type(prompt)}."
              )
          elif batch_size != len(negative_prompt):
              raise ValueError(
                  f"`negative_prompt`: {negative_prompt} has batch size {len(negative_prompt)}, but `prompt`:"
                  f" {prompt} has batch size {batch_size}. Please make sure that passed `negative_prompt` matches"
                  " the batch size of `prompt`."
              )
          else:
              uncond_tokens = [negative_prompt, negative_prompt_2]
  
          negative_prompt_embeds_list = []
          for negative_prompt, tokenizer, text_encoder in zip(
              uncond_tokens, tokenizers, text_encoders
          ):
              if isinstance(self, TextualInversionLoaderMixin):
                  negative_prompt = self.maybe_convert_prompt(
                      negative_prompt, tokenizer
                  )
  
              max_length = prompt_embeds.shape[1]
              uncond_input = tokenizer(
                  negative_prompt,
                  padding="max_length",
                  max_length=max_length,
                  truncation=True,
                  return_tensors="pt",
              )
  
              negative_prompt_embeds = text_encoder(
                  uncond_input.input_ids.to(device),
                  output_hidden_states=True,
              )
              # We are only ALWAYS interested in the pooled output of the final text encoder
              negative_pooled_prompt_embeds = negative_prompt_embeds[0]
              negative_prompt_embeds = (
                  negative_prompt_embeds.hidden_states[-2]
              )
  
              negative_prompt_embeds_list.append(negative_prompt_embeds)
  
          negative_prompt_embeds = torch.concat(
              negative_prompt_embeds_list, dim=-1
          )
  
      if self.text_encoder_2 is not None:
          prompt_embeds = prompt_embeds.to(
              dtype=self.text_encoder_2.dtype, device=device
          )
      else:
          prompt_embeds = prompt_embeds.to(
              dtype=self.unet.dtype, device=device
          )
  
      bs_embed, seq_len, _ = prompt_embeds.shape
      # duplicate text embeddings for each generation per prompt, using mps friendly method
      prompt_embeds = prompt_embeds.repeat(1, num_images_per_prompt, 1)
      prompt_embeds = prompt_embeds.view(
          bs_embed * num_images_per_prompt, seq_len, -1
      )
  
      if do_classifier_free_guidance:
          # duplicate unconditional embeddings for each generation per prompt, using mps friendly method
          seq_len = negative_prompt_embeds.shape[1]
  
          if self.text_encoder_2 is not None:
              negative_prompt_embeds = negative_prompt_embeds.to(
                  dtype=self.text_encoder_2.dtype, device=device
              )
          else:
              negative_prompt_embeds = negative_prompt_embeds.to(
                  dtype=self.unet.dtype, device=device
              )
  
          negative_prompt_embeds = negative_prompt_embeds.repeat(
              1, num_images_per_prompt, 1
          )
          negative_prompt_embeds = negative_prompt_embeds.view(
              batch_size * num_images_per_prompt, seq_len, -1
          )
  
      pooled_prompt_embeds = pooled_prompt_embeds.repeat(
          1, num_images_per_prompt
      ).view(bs_embed * num_images_per_prompt, -1)
      if do_classifier_free_guidance:
          negative_pooled_prompt_embeds = (
              negative_pooled_prompt_embeds.repeat(
                  1, num_images_per_prompt
              ).view(bs_embed * num_images_per_prompt, -1)
          )
  
      if self.text_encoder is not None:
          if (
              isinstance(self, StableDiffusionXLLoraLoaderMixin)
              and USE_PEFT_BACKEND
          ):
              # Retrieve the original scale by scaling back the LoRA layers
              unscale_lora_layers(self.text_encoder, lora_scale)
  
      if self.text_encoder_2 is not None:
          if (
              isinstance(self, StableDiffusionXLLoraLoaderMixin)
              and USE_PEFT_BACKEND
          ):
              # Retrieve the original scale by scaling back the LoRA layers
              unscale_lora_layers(self.text_encoder_2, lora_scale)
  
      return (
          prompt_embeds,
          negative_prompt_embeds,
          pooled_prompt_embeds,
          negative_pooled_prompt_embeds,
      )
  
imports: import inspect
  from typing import Any, Callable, Dict, List, Optional, Tuple, Union
  import numpy
  import PIL.Image
  import torch
  import torch.nn.functional
  from transformers import CLIPTextModel, CLIPTextModelWithProjection, CLIPTokenizer
  from diffusers.image_processor import PipelineImageInput, VaeImageProcessor
  from diffusers.loaders import FromSingleFileMixin, StableDiffusionXLLoraLoaderMixin, TextualInversionLoaderMixin
  from diffusers.models import AutoencoderKL, ControlNetXSModel, UNet2DConditionModel
  from diffusers.models.attention_processor import AttnProcessor2_0, LoRAAttnProcessor2_0, LoRAXFormersAttnProcessor, XFormersAttnProcessor
  from diffusers.models.lora import adjust_lora_scale_text_encoder
  from diffusers.pipelines.pipeline_utils import DiffusionPipeline
  from diffusers.pipelines.stable_diffusion_xl.pipeline_output import StableDiffusionXLPipelineOutput
  from diffusers.schedulers import KarrasDiffusionSchedulers
  from diffusers.utils import USE_PEFT_BACKEND, logging, scale_lora_layers, unscale_lora_layers
  from diffusers.utils.import_utils import is_invisible_watermark_available
  from diffusers.utils.torch_utils import is_compiled_module, is_torch_version, randn_tensor
label_desc: Use image node_id 6 for calling the StableDiffusionXLControlNetXSPipeline obj's underlying member method code with example usage: obj.encode_prompt(prompt, prompt_2, device, num_images_per_prompt, do_classifier_free_guidance, negative_prompt, negative_prompt_2, prompt_embeds, negative_prompt_embeds, pooled_prompt_embeds, negative_pooled_prompt_embeds, lora_scale, clip_skip) and returns: prompt_embeds,  negative_prompt_embeds,  pooled_prompt_embeds,  negative_pooled_prompt_embeds