docstr: Function invoked when calling the pipeline for generation.
  
  Args:
      prompt (`Optional[Union[str, List[str]]]`, defaults to None):
          The prompt or prompts to guide the image generation. If not defined, one has to pass `prompt_embeds`.
          instead.
      height (`Optional[int]`, defaults to None):
          The height in pixels of the generated image.
      width (`Optional[int]`, defaults to None):
          The width in pixels of the generated image.
      num_inference_steps (`int`, defaults to 50):
          The number of denoising steps. More denoising steps usually lead to a higher quality image at the
          expense of slower inference.
      guidance_scale (`float`, defaults to 5):
          Guidance scale as defined in [Classifier-Free Diffusion Guidance](https://arxiv.org/abs/2207.12598).
          `guidance_scale` is defined as `w` of equation 2. of [Imagen
          Paper](https://arxiv.org/pdf/2205.11487.pdf). Guidance scale is enabled by setting `guidance_scale >
          1`. Higher guidance scale encourages to generate images that are closely linked to the text `prompt`,
          usually at the expense of lower image quality.
      negative_prompt (`Optional[Union[str, list]]`):
          The prompt or prompts not to guide the image generation. If not defined, one has to pass
          `negative_prompt_embeds`. instead. Ignored when not using guidance (i.e., ignored if `guidance_scale`
          is less than `1`).
      num_images_per_prompt (`int`, defaults to 1):
          The number of images to generate per prompt.
      eta (`float`, defaults to 0.0):
          Corresponds to parameter eta (η) in the DDIM paper: https://arxiv.org/abs/2010.02502. Only applies to
          [`schedulers.DDIMScheduler`], will be ignored for others.
      generator (`Optional[np.random.RandomState]`, defaults to `None`)::
          A np.random.RandomState to make generation deterministic.
      latents (`Optional[np.ndarray]`, defaults to `None`):
          Pre-generated noisy latents, sampled from a Gaussian distribution, to be used as inputs for image
          generation. Can be used to tweak the same generation with different prompts. If not provided, a latents
          tensor will ge generated by sampling using the supplied random `generator`.
      prompt_embeds (`Optional[np.ndarray]`, defaults to `None`):
          Pre-generated text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not
          provided, text embeddings will be generated from `prompt` input argument.
      negative_prompt_embeds (`Optional[np.ndarray]`, defaults to `None`):
          Pre-generated negative text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt
          weighting. If not provided, negative_prompt_embeds will be generated from `negative_prompt` input
          argument.
      output_type (`str`, defaults to `"pil"`):
          The output format of the generate image. Choose between
          [PIL](https://pillow.readthedocs.io/en/stable/): `PIL.Image.Image` or `np.array`.
      return_dict (`bool`, defaults to `True`):
          Whether or not to return a [`~pipelines.stable_diffusion.StableDiffusionXLPipelineOutput`] instead of a
          plain tuple.
      callback (Optional[Callable], defaults to `None`):
          A function that will be called every `callback_steps` steps during inference. The function will be
          called with the following arguments: `callback(step: int, timestep: int, latents: torch.FloatTensor)`.
      callback_steps (`int`, defaults to 1):
          The frequency at which the `callback` function will be called. If not specified, the callback will be
          called at every step.
      guidance_rescale (`float`, defaults to 0.7):
          Guidance rescale factor proposed by [Common Diffusion Noise Schedules and Sample Steps are
          Flawed](https://arxiv.org/pdf/2305.08891.pdf) `guidance_scale` is defined as `φ` in equation 16. of
          [Common Diffusion Noise Schedules and Sample Steps are Flawed](https://arxiv.org/pdf/2305.08891.pdf).
          Guidance rescale factor should fix overexposure when using zero terminal SNR.
  
  Returns:
      [`~pipelines.stable_diffusion.StableDiffusionXLPipelineOutput`] or `tuple`:
      [`~pipelines.stable_diffusion.StableDiffusionXLPipelineOutput`] if `return_dict` is True, otherwise a `tuple.
      When returning a tuple, the first element is a list with the generated images, and the second element is a
      list of `bool`s denoting whether the corresponding generated image likely represents "not-safe-for-work"
      (nsfw) content, according to the `safety_checker`.
docstr_tok: ["Function","invoked","when","calling","the","pipeline","for","generation",".","Args",":","prompt","(","`","Optional","[","Union","[","str",",","List","[","str","]","]","]","`",",","defaults","to","None",")",":","The","prompt","or","prompts","to","guide","the","image","generation",".","If","not","defined",",","one","has","to","pass","`","prompt_embeds","`",".","instead",".","height","(","`","Optional","[","int","]","`",",","defaults","to","None",")",":","The","height","in","pixels","of","the","generated","image",".","width","(","`","Optional","[","int","]","`",",","defaults","to","None",")",":","The","width","in","pixels","of","the","generated","image",".","num_inference_steps","(","`","int","`",",","defaults","to","50",")",":","The","number","of","denoising","steps",".","More","denoising","steps","usually","lead","to","a","higher","quality","image","at","the","expense","of","slower","inference",".","guidance_scale","(","`","float","`",",","defaults","to","5",")",":","Guidance","scale","as","defined","in","[","Classifier-Free","Diffusion","Guidance","]","(","https",":","\/\/arxiv.org\/abs\/2207.12598",")",".","`","guidance_scale","`","is","defined","as","`","w","`","of","equation","2.","of","[","Imagen","Paper","]","(","https",":","\/\/arxiv.org\/pdf\/2205.11487.pdf",")",".","Guidance","scale","is","enabled","by","setting","`","guidance_scale",">","1","`",".","Higher","guidance","scale","encourages","to","generate","images","that","are","closely","linked","to","the","text","`","prompt","`",",","usually","at","the","expense","of","lower","image","quality",".","negative_prompt","(","`","Optional","[","Union","[","str",",","list","]","]","`",")",":","The","prompt","or","prompts","not","to","guide","the","image","generation",".","If","not","defined",",","one","has","to","pass","`","negative_prompt_embeds","`",".","instead",".","Ignored","when","not","using","guidance","(","i.e.",",","ignored","if","`","guidance_scale","`","is","less","than","`","1","`",")",".","num_images_per_prompt","(","`","int","`",",","defaults","to","1",")",":","The","number","of","images","to","generate","per","prompt",".","eta","(","`","float","`",",","defaults","to","0.0",")",":","Corresponds","to","parameter","eta","(","\u03b7",")","in","the","DDIM","paper",":","https",":","\/\/arxiv.org\/abs\/2010.02502",".","Only","applies","to","[","`","schedulers.DDIMScheduler","`","]",",","will","be","ignored","for","others",".","generator","(","`","Optional","[","np.random.RandomState","]","`",",","defaults","to","`","None","`",")",":",":","A","np.random.RandomState","to","make","generation","deterministic",".","latents","(","`","Optional","[","np.ndarray","]","`",",","defaults","to","`","None","`",")",":","Pre-generated","noisy","latents",",","sampled","from","a","Gaussian","distribution",",","to","be","used","as","inputs","for","image","generation",".","Can","be","used","to","tweak","the","same","generation","with","different","prompts",".","If","not","provided",",","a","latents","tensor","will","ge","generated","by","sampling","using","the","supplied","random","`","generator","`",".","prompt_embeds","(","`","Optional","[","np.ndarray","]","`",",","defaults","to","`","None","`",")",":","Pre-generated","text","embeddings",".","Can","be","used","to","easily","tweak","text","inputs",",","*","e.g",".","*","prompt","weighting",".","If","not","provided",",","text","embeddings","will","be","generated","from","`","prompt","`","input","argument",".","negative_prompt_embeds","(","`","Optional","[","np.ndarray","]","`",",","defaults","to","`","None","`",")",":","Pre-generated","negative","text","embeddings",".","Can","be","used","to","easily","tweak","text","inputs",",","*","e.g",".","*","prompt","weighting",".","If","not","provided",",","negative_prompt_embeds","will","be","generated","from","`","negative_prompt","`","input","argument",".","output_type","(","`","str","`",",","defaults","to","`","``","pil","''","`",")",":","The","output","format","of","the","generate","image",".","Choose","between","[","PIL","]","(","https",":","\/\/pillow.readthedocs.io\/en\/stable\/",")",":","`","PIL.Image.Image","`","or","`","np.array","`",".","return_dict","(","`","bool","`",",","defaults","to","`","True","`",")",":","Whether","or","not","to","return","a","[","`","~pipelines.stable_diffusion.StableDiffusionXLPipelineOutput","`","]","instead","of","a","plain","tuple",".","callback","(","Optional","[","Callable","]",",","defaults","to","`","None","`",")",":","A","function","that","will","be","called","every","`","callback_steps","`","steps","during","inference",".","The","function","will","be","called","with","the","following","arguments",":","`","callback","(","step",":","int",",","timestep",":","int",",","latents",":","torch.FloatTensor",")","`",".","callback_steps","(","`","int","`",",","defaults","to","1",")",":","The","frequency","at","which","the","`","callback","`","function","will","be","called",".","If","not","specified",",","the","callback","will","be","called","at","every","step",".","guidance_rescale","(","`","float","`",",","defaults","to","0.7",")",":","Guidance","rescale","factor","proposed","by","[","Common","Diffusion","Noise","Schedules","and","Sample","Steps","are","Flawed","]","(","https",":","\/\/arxiv.org\/pdf\/2305.08891.pdf",")","`","guidance_scale","`","is","defined","as","`","\u03c6","`","in","equation","16.","of","[","Common","Diffusion","Noise","Schedules","and","Sample","Steps","are","Flawed","]","(","https",":","\/\/arxiv.org\/pdf\/2305.08891.pdf",")",".","Guidance","rescale","factor","should","fix","overexposure","when","using","zero","terminal","SNR",".","Returns",":","[","`","~pipelines.stable_diffusion.StableDiffusionXLPipelineOutput","`","]","or","`","tuple","`",":","[","`","~pipelines.stable_diffusion.StableDiffusionXLPipelineOutput","`","]","if","`","return_dict","`","is","True",",","otherwise","a","`","tuple",".","When","returning","a","tuple",",","the","first","element","is","a","list","with","the","generated","images",",","and","the","second","element","is","a","list","of","`","bool","`","s","denoting","whether","the","corresponding","generated","image","likely","represents","``","not-safe-for-work","''","(","nsfw",")","content",",","according","to","the","`","safety_checker","`","."]
code: def __call__(
      self,
      prompt: Optional[Union[str, List[str]]] = None,
      height: Optional[int] = None,
      width: Optional[int] = None,
      num_inference_steps: int = 50,
      guidance_scale: float = 5.0,
      negative_prompt: Optional[Union[str, List[str]]] = None,
      num_images_per_prompt: int = 1,
      eta: float = 0.0,
      generator: Optional[np.random.RandomState] = None,
      latents: Optional[np.ndarray] = None,
      prompt_embeds: Optional[np.ndarray] = None,
      negative_prompt_embeds: Optional[np.ndarray] = None,
      pooled_prompt_embeds: Optional[np.ndarray] = None,
      negative_pooled_prompt_embeds: Optional[np.ndarray] = None,
      output_type: str = "pil",
      return_dict: bool = True,
      callback: Optional[Callable[[int, int, np.ndarray], None]] = None,
      callback_steps: int = 1,
      cross_attention_kwargs: Optional[Dict[str, Any]] = None,
      guidance_rescale: float = 0.0,
      original_size: Optional[Tuple[int, int]] = None,
      crops_coords_top_left: Tuple[int, int] = (0, 0),
      target_size: Optional[Tuple[int, int]] = None,
  ):
      r"""
      Function invoked when calling the pipeline for generation.
  
      Args:
          prompt (`Optional[Union[str, List[str]]]`, defaults to None):
              The prompt or prompts to guide the image generation. If not defined, one has to pass `prompt_embeds`.
              instead.
          height (`Optional[int]`, defaults to None):
              The height in pixels of the generated image.
          width (`Optional[int]`, defaults to None):
              The width in pixels of the generated image.
          num_inference_steps (`int`, defaults to 50):
              The number of denoising steps. More denoising steps usually lead to a higher quality image at the
              expense of slower inference.
          guidance_scale (`float`, defaults to 5):
              Guidance scale as defined in [Classifier-Free Diffusion Guidance](https://arxiv.org/abs/2207.12598).
              `guidance_scale` is defined as `w` of equation 2. of [Imagen
              Paper](https://arxiv.org/pdf/2205.11487.pdf). Guidance scale is enabled by setting `guidance_scale >
              1`. Higher guidance scale encourages to generate images that are closely linked to the text `prompt`,
              usually at the expense of lower image quality.
          negative_prompt (`Optional[Union[str, list]]`):
              The prompt or prompts not to guide the image generation. If not defined, one has to pass
              `negative_prompt_embeds`. instead. Ignored when not using guidance (i.e., ignored if `guidance_scale`
              is less than `1`).
          num_images_per_prompt (`int`, defaults to 1):
              The number of images to generate per prompt.
          eta (`float`, defaults to 0.0):
              Corresponds to parameter eta (η) in the DDIM paper: https://arxiv.org/abs/2010.02502. Only applies to
              [`schedulers.DDIMScheduler`], will be ignored for others.
          generator (`Optional[np.random.RandomState]`, defaults to `None`)::
              A np.random.RandomState to make generation deterministic.
          latents (`Optional[np.ndarray]`, defaults to `None`):
              Pre-generated noisy latents, sampled from a Gaussian distribution, to be used as inputs for image
              generation. Can be used to tweak the same generation with different prompts. If not provided, a latents
              tensor will ge generated by sampling using the supplied random `generator`.
          prompt_embeds (`Optional[np.ndarray]`, defaults to `None`):
              Pre-generated text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not
              provided, text embeddings will be generated from `prompt` input argument.
          negative_prompt_embeds (`Optional[np.ndarray]`, defaults to `None`):
              Pre-generated negative text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt
              weighting. If not provided, negative_prompt_embeds will be generated from `negative_prompt` input
              argument.
          output_type (`str`, defaults to `"pil"`):
              The output format of the generate image. Choose between
              [PIL](https://pillow.readthedocs.io/en/stable/): `PIL.Image.Image` or `np.array`.
          return_dict (`bool`, defaults to `True`):
              Whether or not to return a [`~pipelines.stable_diffusion.StableDiffusionXLPipelineOutput`] instead of a
              plain tuple.
          callback (Optional[Callable], defaults to `None`):
              A function that will be called every `callback_steps` steps during inference. The function will be
              called with the following arguments: `callback(step: int, timestep: int, latents: torch.FloatTensor)`.
          callback_steps (`int`, defaults to 1):
              The frequency at which the `callback` function will be called. If not specified, the callback will be
              called at every step.
          guidance_rescale (`float`, defaults to 0.7):
              Guidance rescale factor proposed by [Common Diffusion Noise Schedules and Sample Steps are
              Flawed](https://arxiv.org/pdf/2305.08891.pdf) `guidance_scale` is defined as `φ` in equation 16. of
              [Common Diffusion Noise Schedules and Sample Steps are Flawed](https://arxiv.org/pdf/2305.08891.pdf).
              Guidance rescale factor should fix overexposure when using zero terminal SNR.
  
      Returns:
          [`~pipelines.stable_diffusion.StableDiffusionXLPipelineOutput`] or `tuple`:
          [`~pipelines.stable_diffusion.StableDiffusionXLPipelineOutput`] if `return_dict` is True, otherwise a `tuple.
          When returning a tuple, the first element is a list with the generated images, and the second element is a
          list of `bool`s denoting whether the corresponding generated image likely represents "not-safe-for-work"
          (nsfw) content, according to the `safety_checker`.
      """
  
      # 0. Default height and width to unet
      height = (
          height
          or self.unet.config["sample_size"] * self.vae_scale_factor
      )
      width = (
          width
          or self.unet.config["sample_size"] * self.vae_scale_factor
      )
  
      original_size = original_size or (height, width)
      target_size = target_size or (height, width)
  
      # 1. Check inputs. Raise error if not correct
      self.check_inputs(
          prompt,
          height,
          width,
          callback_steps,
          negative_prompt,
          prompt_embeds,
          negative_prompt_embeds,
          pooled_prompt_embeds,
          negative_pooled_prompt_embeds,
      )
  
      # 2. Define call parameters
      if isinstance(prompt, str):
          batch_size = 1
      elif isinstance(prompt, list):
          batch_size = len(prompt)
      else:
          batch_size = prompt_embeds.shape[0]
  
      if generator is None:
          generator = np.random
  
      # here `guidance_scale` is defined analog to the guidance weight `w` of equation (2)
      # of the Imagen paper: https://arxiv.org/pdf/2205.11487.pdf . `guidance_scale = 1`
      # corresponds to doing no classifier free guidance.
      do_classifier_free_guidance = guidance_scale > 1.0
  
      # 3. Encode input prompt
      (
          prompt_embeds,
          negative_prompt_embeds,
          pooled_prompt_embeds,
          negative_pooled_prompt_embeds,
      ) = self._encode_prompt(
          prompt,
          num_images_per_prompt,
          do_classifier_free_guidance,
          negative_prompt,
          prompt_embeds=prompt_embeds,
          negative_prompt_embeds=negative_prompt_embeds,
          pooled_prompt_embeds=pooled_prompt_embeds,
          negative_pooled_prompt_embeds=negative_pooled_prompt_embeds,
      )
  
      # 4. Prepare timesteps
      self.scheduler.set_timesteps(num_inference_steps)
      timesteps = self.scheduler.timesteps
  
      # 5. Prepare latent variables
      latents = self.prepare_latents(
          batch_size * num_images_per_prompt,
          self.unet.config.get("in_channels", 4),
          height,
          width,
          prompt_embeds.dtype,
          generator,
          latents,
      )
  
      # 6. Prepare extra step kwargs
      extra_step_kwargs = self.prepare_extra_step_kwargs(generator, eta)
  
      # 7. Prepare added time ids & embeddings
      add_text_embeds = pooled_prompt_embeds
      add_time_ids = (
          original_size + crops_coords_top_left + target_size,
      )
      add_time_ids = np.array(add_time_ids, dtype=prompt_embeds.dtype)
  
      if do_classifier_free_guidance:
          prompt_embeds = np.concatenate(
              (negative_prompt_embeds, prompt_embeds), axis=0
          )
          add_text_embeds = np.concatenate(
              (negative_pooled_prompt_embeds, add_text_embeds), axis=0
          )
          add_time_ids = np.concatenate(
              (add_time_ids, add_time_ids), axis=0
          )
      add_time_ids = np.repeat(
          add_time_ids, batch_size * num_images_per_prompt, axis=0
      )
  
      # Adapted from diffusers to extend it for other runtimes than ORT
      timestep_dtype = self.unet.input_dtype.get("timestep", np.float32)
  
      # 8. Denoising loop
      num_warmup_steps = (
          len(timesteps) - num_inference_steps * self.scheduler.order
      )
      for i, t in enumerate(self.progress_bar(timesteps)):
          # expand the latents if we are doing classifier free guidance
          latent_model_input = (
              np.concatenate([latents] * 2)
              if do_classifier_free_guidance
              else latents
          )
          latent_model_input = self.scheduler.scale_model_input(
              torch.from_numpy(latent_model_input), t
          )
          latent_model_input = latent_model_input.cpu().numpy()
  
          # predict the noise residual
          timestep = np.array([t], dtype=timestep_dtype)
          noise_pred = self.unet(
              sample=latent_model_input,
              timestep=timestep,
              encoder_hidden_states=prompt_embeds,
              text_embeds=add_text_embeds,
              time_ids=add_time_ids,
          )
          noise_pred = noise_pred[0]
  
          # perform guidance
          if do_classifier_free_guidance:
              noise_pred_uncond, noise_pred_text = np.split(
                  noise_pred, 2
              )
              noise_pred = noise_pred_uncond + guidance_scale * (
                  noise_pred_text - noise_pred_uncond
              )
              if guidance_rescale > 0.0:
                  # Based on 3.4. in https://arxiv.org/pdf/2305.08891.pdf
                  noise_pred = rescale_noise_cfg(
                      noise_pred,
                      noise_pred_text,
                      guidance_rescale=guidance_rescale,
                  )
  
          # compute the previous noisy sample x_t -> x_t-1
          scheduler_output = self.scheduler.step(
              torch.from_numpy(noise_pred),
              t,
              torch.from_numpy(latents),
              **extra_step_kwargs,
          )
          latents = scheduler_output.prev_sample.numpy()
  
          # call the callback, if provided
          if i == len(timesteps) - 1 or (
              (i + 1) > num_warmup_steps
              and (i + 1) % self.scheduler.order == 0
          ):
              if callback is not None and i % callback_steps == 0:
                  callback(i, t, latents)
  
      if output_type == "latent":
          image = latents
      else:
          latents /= self.vae_decoder.config.get(
              "scaling_factor", 0.18215
          )
          # it seems likes there is a strange result for using half-precision vae decoder if batchsize>1
          image = np.concatenate(
              [
                  self.vae_decoder(latent_sample=latents[i : i + 1])[0]
                  for i in range(latents.shape[0])
              ]
          )
          # apply watermark if available
          if self.watermark is not None:
              image = self.watermark.apply_watermark(image)
          image = self.image_processor.postprocess(
              image, output_type=output_type
          )
  
      if not return_dict:
          return (image,)
  
      return StableDiffusionXLPipelineOutput(images=image)
  
file_path: optimum/optimum/pipelines/diffusers/pipeline_stable_diffusion_xl.py
imports: import inspect
  import logging
  from typing import Any, Callable, Dict, List, Optional, Tuple, Union
  import numpy
  import torch
  from diffusers.pipelines.stable_diffusion_xl import StableDiffusionXLPipelineOutput
  from .pipeline_utils import DiffusionPipelineMixin, rescale_noise_cfg
label_desc: Use image node_id 5 for calling the StableDiffusionXLPipelineMixin obj's underlying member method code with example usage: obj.__call__(prompt, height, width, num_inference_steps, guidance_scale, negative_prompt, num_images_per_prompt, eta, generator, latents, prompt_embeds, negative_prompt_embeds, pooled_prompt_embeds, negative_pooled_prompt_embeds, output_type, return_dict, callback, callback_steps, cross_attention_kwargs, guidance_rescale, original_size, crops_coords_top_left, target_size) and returns: StableDiffusionXLPipelineOutput, image