class_docstr: Pipeline for a bimodal image-text model which supports unconditional text and image generation, text-conditioned
  image generation, image-conditioned text generation, and joint image-text generation.
  
  This model inherits from [`DiffusionPipeline`]. Check the superclass documentation for the generic methods
  implemented for all pipelines (downloading, saving, running on a particular device, etc.).
  
  Args:
      vae ([`AutoencoderKL`]):
          Variational Auto-Encoder (VAE) model to encode and decode images to and from latent representations. This
          is part of the UniDiffuser image representation along with the CLIP vision encoding.
      text_encoder ([`CLIPTextModel`]):
          Frozen text-encoder ([clip-vit-large-patch14](https://huggingface.co/openai/clip-vit-large-patch14)).
      image_encoder ([`CLIPVisionModel`]):
          A [`~transformers.CLIPVisionModel`] to encode images as part of its image representation along with the VAE
          latent representation.
      image_processor ([`CLIPImageProcessor`]):
          [`~transformers.CLIPImageProcessor`] to preprocess an image before CLIP encoding it with `image_encoder`.
      clip_tokenizer ([`CLIPTokenizer`]):
           A [`~transformers.CLIPTokenizer`] to tokenize the prompt before encoding it with `text_encoder`.
      text_decoder ([`UniDiffuserTextDecoder`]):
          Frozen text decoder. This is a GPT-style model which is used to generate text from the UniDiffuser
          embedding.
      text_tokenizer ([`GPT2Tokenizer`]):
          A [`~transformers.GPT2Tokenizer`] to decode text for text generation; used along with the `text_decoder`.
      unet ([`UniDiffuserModel`]):
          A [U-ViT](https://github.com/baofff/U-ViT) model with UNNet-style skip connections between transformer
          layers to denoise the encoded image latents.
      scheduler ([`SchedulerMixin`]):
          A scheduler to be used in combination with `unet` to denoise the encoded image and/or text latents. The
          original UniDiffuser paper uses the [`DPMSolverMultistepScheduler`] scheduler.
class_docstr_tok: ["Pipeline","for","a","bimodal","image-text","model","which","supports","unconditional","text","and","image","generation",",","text-conditioned","image","generation",",","image-conditioned","text","generation",",","and","joint","image-text","generation",".","This","model","inherits","from","[","`","DiffusionPipeline","`","]",".","Check","the","superclass","documentation","for","the","generic","methods","implemented","for","all","pipelines","(","downloading",",","saving",",","running","on","a","particular","device",",","etc.",")",".","Args",":","vae","(","[","`","AutoencoderKL","`","]",")",":","Variational","Auto-Encoder","(","VAE",")","model","to","encode","and","decode","images","to","and","from","latent","representations",".","This","is","part","of","the","UniDiffuser","image","representation","along","with","the","CLIP","vision","encoding",".","text_encoder","(","[","`","CLIPTextModel","`","]",")",":","Frozen","text-encoder","(","[","clip-vit-large-patch14","]","(","https",":","\/\/huggingface.co\/openai\/clip-vit-large-patch14",")",")",".","image_encoder","(","[","`","CLIPVisionModel","`","]",")",":","A","[","`","~transformers.CLIPVisionModel","`","]","to","encode","images","as","part","of","its","image","representation","along","with","the","VAE","latent","representation",".","image_processor","(","[","`","CLIPImageProcessor","`","]",")",":","[","`","~transformers.CLIPImageProcessor","`","]","to","preprocess","an","image","before","CLIP","encoding","it","with","`","image_encoder","`",".","clip_tokenizer","(","[","`","CLIPTokenizer","`","]",")",":","A","[","`","~transformers.CLIPTokenizer","`","]","to","tokenize","the","prompt","before","encoding","it","with","`","text_encoder","`",".","text_decoder","(","[","`","UniDiffuserTextDecoder","`","]",")",":","Frozen","text","decoder",".","This","is","a","GPT-style","model","which","is","used","to","generate","text","from","the","UniDiffuser","embedding",".","text_tokenizer","(","[","`","GPT2Tokenizer","`","]",")",":","A","[","`","~transformers.GPT2Tokenizer","`","]","to","decode","text","for","text","generation",";","used","along","with","the","`","text_decoder","`",".","unet","(","[","`","UniDiffuserModel","`","]",")",":","A","[","U-ViT","]","(","https",":","\/\/github.com\/baofff\/U-ViT",")","model","with","UNNet-style","skip","connections","between","transformer","layers","to","denoise","the","encoded","image","latents",".","scheduler","(","[","`","SchedulerMixin","`","]",")",":","A","scheduler","to","be","used","in","combination","with","`","unet","`","to","denoise","the","encoded","image","and\/or","text","latents",".","The","original","UniDiffuser","paper","uses","the","[","`","DPMSolverMultistepScheduler","`","]","scheduler","."]
docstr: The call function to the pipeline for generation.
  
  Args:
      prompt (`str` or `List[str]`, *optional*):
          The prompt or prompts to guide image generation. If not defined, you need to pass `prompt_embeds`.
          Required for text-conditioned image generation (`text2img`) mode.
      image (`torch.FloatTensor` or `PIL.Image.Image`, *optional*):
          `Image` or tensor representing an image batch. Required for image-conditioned text generation
          (`img2text`) mode.
      height (`int`, *optional*, defaults to `self.unet.config.sample_size * self.vae_scale_factor`):
          The height in pixels of the generated image.
      width (`int`, *optional*, defaults to `self.unet.config.sample_size * self.vae_scale_factor`):
          The width in pixels of the generated image.
      data_type (`int`, *optional*, defaults to 1):
          The data type (either 0 or 1). Only used if you are loading a checkpoint which supports a data type
          embedding; this is added for compatibility with the
          [UniDiffuser-v1](https://huggingface.co/thu-ml/unidiffuser-v1) checkpoint.
      num_inference_steps (`int`, *optional*, defaults to 50):
          The number of denoising steps. More denoising steps usually lead to a higher quality image at the
          expense of slower inference.
      guidance_scale (`float`, *optional*, defaults to 8.0):
          A higher guidance scale value encourages the model to generate images closely linked to the text
          `prompt` at the expense of lower image quality. Guidance scale is enabled when `guidance_scale > 1`.
      negative_prompt (`str` or `List[str]`, *optional*):
          The prompt or prompts to guide what to not include in image generation. If not defined, you need to
          pass `negative_prompt_embeds` instead. Ignored when not using guidance (`guidance_scale < 1`). Used in
          text-conditioned image generation (`text2img`) mode.
      num_images_per_prompt (`int`, *optional*, defaults to 1):
          The number of images to generate per prompt. Used in `text2img` (text-conditioned image generation) and
          `img` mode. If the mode is joint and both `num_images_per_prompt` and `num_prompts_per_image` are
          supplied, `min(num_images_per_prompt, num_prompts_per_image)` samples are generated.
      num_prompts_per_image (`int`, *optional*, defaults to 1):
          The number of prompts to generate per image. Used in `img2text` (image-conditioned text generation) and
          `text` mode. If the mode is joint and both `num_images_per_prompt` and `num_prompts_per_image` are
          supplied, `min(num_images_per_prompt, num_prompts_per_image)` samples are generated.
      eta (`float`, *optional*, defaults to 0.0):
          Corresponds to parameter eta (η) from the [DDIM](https://arxiv.org/abs/2010.02502) paper. Only applies
          to the [`~schedulers.DDIMScheduler`], and is ignored in other schedulers.
      generator (`torch.Generator` or `List[torch.Generator]`, *optional*):
          A [`torch.Generator`](https://pytorch.org/docs/stable/generated/torch.Generator.html) to make
          generation deterministic.
      latents (`torch.FloatTensor`, *optional*):
          Pre-generated noisy latents sampled from a Gaussian distribution, to be used as inputs for joint
          image-text generation. Can be used to tweak the same generation with different prompts. If not
          provided, a latents tensor is generated by sampling using the supplied random `generator`. This assumes
          a full set of VAE, CLIP, and text latents, if supplied, overrides the value of `prompt_latents`,
          `vae_latents`, and `clip_latents`.
      prompt_latents (`torch.FloatTensor`, *optional*):
          Pre-generated noisy latents sampled from a Gaussian distribution, to be used as inputs for text
          generation. Can be used to tweak the same generation with different prompts. If not provided, a latents
          tensor is generated by sampling using the supplied random `generator`.
      vae_latents (`torch.FloatTensor`, *optional*):
          Pre-generated noisy latents sampled from a Gaussian distribution, to be used as inputs for image
          generation. Can be used to tweak the same generation with different prompts. If not provided, a latents
          tensor is generated by sampling using the supplied random `generator`.
      clip_latents (`torch.FloatTensor`, *optional*):
          Pre-generated noisy latents sampled from a Gaussian distribution, to be used as inputs for image
          generation. Can be used to tweak the same generation with different prompts. If not provided, a latents
          tensor is generated by sampling using the supplied random `generator`.
      prompt_embeds (`torch.FloatTensor`, *optional*):
          Pre-generated text embeddings. Can be used to easily tweak text inputs (prompt weighting). If not
          provided, text embeddings are generated from the `prompt` input argument. Used in text-conditioned
          image generation (`text2img`) mode.
      negative_prompt_embeds (`torch.FloatTensor`, *optional*):
          Pre-generated negative text embeddings. Can be used to easily tweak text inputs (prompt weighting). If
          not provided, `negative_prompt_embeds` are be generated from the `negative_prompt` input argument. Used
          in text-conditioned image generation (`text2img`) mode.
      output_type (`str`, *optional*, defaults to `"pil"`):
          The output format of the generated image. Choose between `PIL.Image` or `np.array`.
      return_dict (`bool`, *optional*, defaults to `True`):
          Whether or not to return a [`~pipelines.ImageTextPipelineOutput`] instead of a plain tuple.
      callback (`Callable`, *optional*):
          A function that calls every `callback_steps` steps during inference. The function is called with the
          following arguments: `callback(step: int, timestep: int, latents: torch.FloatTensor)`.
      callback_steps (`int`, *optional*, defaults to 1):
          The frequency at which the `callback` function is called. If not specified, the callback is called at
          every step.
  
  Returns:
      [`~pipelines.unidiffuser.ImageTextPipelineOutput`] or `tuple`:
          If `return_dict` is `True`, [`~pipelines.unidiffuser.ImageTextPipelineOutput`] is returned, otherwise a
          `tuple` is returned where the first element is a list with the generated images and the second element
          is a list of generated texts.
docstr_tok: ["The","call","function","to","the","pipeline","for","generation",".","Args",":","prompt","(","`","str","`","or","`","List","[","str","]","`",",","*","optional","*",")",":","The","prompt","or","prompts","to","guide","image","generation",".","If","not","defined",",","you","need","to","pass","`","prompt_embeds","`",".","Required","for","text-conditioned","image","generation","(","`","text2img","`",")","mode",".","image","(","`","torch.FloatTensor","`","or","`","PIL.Image.Image","`",",","*","optional","*",")",":","`","Image","`","or","tensor","representing","an","image","batch",".","Required","for","image-conditioned","text","generation","(","`","img2text","`",")","mode",".","height","(","`","int","`",",","*","optional","*",",","defaults","to","`","self.unet.config.sample_size","*","self.vae_scale_factor","`",")",":","The","height","in","pixels","of","the","generated","image",".","width","(","`","int","`",",","*","optional","*",",","defaults","to","`","self.unet.config.sample_size","*","self.vae_scale_factor","`",")",":","The","width","in","pixels","of","the","generated","image",".","data_type","(","`","int","`",",","*","optional","*",",","defaults","to","1",")",":","The","data","type","(","either","0","or","1",")",".","Only","used","if","you","are","loading","a","checkpoint","which","supports","a","data","type","embedding",";","this","is","added","for","compatibility","with","the","[","UniDiffuser-v1","]","(","https",":","\/\/huggingface.co\/thu-ml\/unidiffuser-v1",")","checkpoint",".","num_inference_steps","(","`","int","`",",","*","optional","*",",","defaults","to","50",")",":","The","number","of","denoising","steps",".","More","denoising","steps","usually","lead","to","a","higher","quality","image","at","the","expense","of","slower","inference",".","guidance_scale","(","`","float","`",",","*","optional","*",",","defaults","to","8.0",")",":","A","higher","guidance","scale","value","encourages","the","model","to","generate","images","closely","linked","to","the","text","`","prompt","`","at","the","expense","of","lower","image","quality",".","Guidance","scale","is","enabled","when","`","guidance_scale",">","1","`",".","negative_prompt","(","`","str","`","or","`","List","[","str","]","`",",","*","optional","*",")",":","The","prompt","or","prompts","to","guide","what","to","not","include","in","image","generation",".","If","not","defined",",","you","need","to","pass","`","negative_prompt_embeds","`","instead",".","Ignored","when","not","using","guidance","(","`","guidance_scale","<","1","`",")",".","Used","in","text-conditioned","image","generation","(","`","text2img","`",")","mode",".","num_images_per_prompt","(","`","int","`",",","*","optional","*",",","defaults","to","1",")",":","The","number","of","images","to","generate","per","prompt",".","Used","in","`","text2img","`","(","text-conditioned","image","generation",")","and","`","img","`","mode",".","If","the","mode","is","joint","and","both","`","num_images_per_prompt","`","and","`","num_prompts_per_image","`","are","supplied",",","`","min","(","num_images_per_prompt",",","num_prompts_per_image",")","`","samples","are","generated",".","num_prompts_per_image","(","`","int","`",",","*","optional","*",",","defaults","to","1",")",":","The","number","of","prompts","to","generate","per","image",".","Used","in","`","img2text","`","(","image-conditioned","text","generation",")","and","`","text","`","mode",".","If","the","mode","is","joint","and","both","`","num_images_per_prompt","`","and","`","num_prompts_per_image","`","are","supplied",",","`","min","(","num_images_per_prompt",",","num_prompts_per_image",")","`","samples","are","generated",".","eta","(","`","float","`",",","*","optional","*",",","defaults","to","0.0",")",":","Corresponds","to","parameter","eta","(","\u03b7",")","from","the","[","DDIM","]","(","https",":","\/\/arxiv.org\/abs\/2010.02502",")","paper",".","Only","applies","to","the","[","`","~schedulers.DDIMScheduler","`","]",",","and","is","ignored","in","other","schedulers",".","generator","(","`","torch.Generator","`","or","`","List","[","torch.Generator","]","`",",","*","optional","*",")",":","A","[","`","torch.Generator","`","]","(","https",":","\/\/pytorch.org\/docs\/stable\/generated\/torch.Generator.html",")","to","make","generation","deterministic",".","latents","(","`","torch.FloatTensor","`",",","*","optional","*",")",":","Pre-generated","noisy","latents","sampled","from","a","Gaussian","distribution",",","to","be","used","as","inputs","for","joint","image-text","generation",".","Can","be","used","to","tweak","the","same","generation","with","different","prompts",".","If","not","provided",",","a","latents","tensor","is","generated","by","sampling","using","the","supplied","random","`","generator","`",".","This","assumes","a","full","set","of","VAE",",","CLIP",",","and","text","latents",",","if","supplied",",","overrides","the","value","of","`","prompt_latents","`",",","`","vae_latents","`",",","and","`","clip_latents","`",".","prompt_latents","(","`","torch.FloatTensor","`",",","*","optional","*",")",":","Pre-generated","noisy","latents","sampled","from","a","Gaussian","distribution",",","to","be","used","as","inputs","for","text","generation",".","Can","be","used","to","tweak","the","same","generation","with","different","prompts",".","If","not","provided",",","a","latents","tensor","is","generated","by","sampling","using","the","supplied","random","`","generator","`",".","vae_latents","(","`","torch.FloatTensor","`",",","*","optional","*",")",":","Pre-generated","noisy","latents","sampled","from","a","Gaussian","distribution",",","to","be","used","as","inputs","for","image","generation",".","Can","be","used","to","tweak","the","same","generation","with","different","prompts",".","If","not","provided",",","a","latents","tensor","is","generated","by","sampling","using","the","supplied","random","`","generator","`",".","clip_latents","(","`","torch.FloatTensor","`",",","*","optional","*",")",":","Pre-generated","noisy","latents","sampled","from","a","Gaussian","distribution",",","to","be","used","as","inputs","for","image","generation",".","Can","be","used","to","tweak","the","same","generation","with","different","prompts",".","If","not","provided",",","a","latents","tensor","is","generated","by","sampling","using","the","supplied","random","`","generator","`",".","prompt_embeds","(","`","torch.FloatTensor","`",",","*","optional","*",")",":","Pre-generated","text","embeddings",".","Can","be","used","to","easily","tweak","text","inputs","(","prompt","weighting",")",".","If","not","provided",",","text","embeddings","are","generated","from","the","`","prompt","`","input","argument",".","Used","in","text-conditioned","image","generation","(","`","text2img","`",")","mode",".","negative_prompt_embeds","(","`","torch.FloatTensor","`",",","*","optional","*",")",":","Pre-generated","negative","text","embeddings",".","Can","be","used","to","easily","tweak","text","inputs","(","prompt","weighting",")",".","If","not","provided",",","`","negative_prompt_embeds","`","are","be","generated","from","the","`","negative_prompt","`","input","argument",".","Used","in","text-conditioned","image","generation","(","`","text2img","`",")","mode",".","output_type","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","pil","''","`",")",":","The","output","format","of","the","generated","image",".","Choose","between","`","PIL.Image","`","or","`","np.array","`",".","return_dict","(","`","bool","`",",","*","optional","*",",","defaults","to","`","True","`",")",":","Whether","or","not","to","return","a","[","`","~pipelines.ImageTextPipelineOutput","`","]","instead","of","a","plain","tuple",".","callback","(","`","Callable","`",",","*","optional","*",")",":","A","function","that","calls","every","`","callback_steps","`","steps","during","inference",".","The","function","is","called","with","the","following","arguments",":","`","callback","(","step",":","int",",","timestep",":","int",",","latents",":","torch.FloatTensor",")","`",".","callback_steps","(","`","int","`",",","*","optional","*",",","defaults","to","1",")",":","The","frequency","at","which","the","`","callback","`","function","is","called",".","If","not","specified",",","the","callback","is","called","at","every","step",".","Returns",":","[","`","~pipelines.unidiffuser.ImageTextPipelineOutput","`","]","or","`","tuple","`",":","If","`","return_dict","`","is","`","True","`",",","[","`","~pipelines.unidiffuser.ImageTextPipelineOutput","`","]","is","returned",",","otherwise","a","`","tuple","`","is","returned","where","the","first","element","is","a","list","with","the","generated","images","and","the","second","element","is","a","list","of","generated","texts","."]
code: def __call__(
      self,
      prompt: Optional[Union[str, List[str]]] = None,
      image: Optional[Union[torch.FloatTensor, PIL.Image.Image]] = None,
      height: Optional[int] = None,
      width: Optional[int] = None,
      data_type: Optional[int] = 1,
      num_inference_steps: int = 50,
      guidance_scale: float = 8.0,
      negative_prompt: Optional[Union[str, List[str]]] = None,
      num_images_per_prompt: Optional[int] = 1,
      num_prompts_per_image: Optional[int] = 1,
      eta: float = 0.0,
      generator: Optional[
          Union[torch.Generator, List[torch.Generator]]
      ] = None,
      latents: Optional[torch.FloatTensor] = None,
      prompt_latents: Optional[torch.FloatTensor] = None,
      vae_latents: Optional[torch.FloatTensor] = None,
      clip_latents: Optional[torch.FloatTensor] = None,
      prompt_embeds: Optional[torch.FloatTensor] = None,
      negative_prompt_embeds: Optional[torch.FloatTensor] = None,
      output_type: Optional[str] = "pil",
      return_dict: bool = True,
      callback: Optional[
          Callable[[int, int, torch.FloatTensor], None]
      ] = None,
      callback_steps: int = 1,
  ):
      r"""
      The call function to the pipeline for generation.
  
      Args:
          prompt (`str` or `List[str]`, *optional*):
              The prompt or prompts to guide image generation. If not defined, you need to pass `prompt_embeds`.
              Required for text-conditioned image generation (`text2img`) mode.
          image (`torch.FloatTensor` or `PIL.Image.Image`, *optional*):
              `Image` or tensor representing an image batch. Required for image-conditioned text generation
              (`img2text`) mode.
          height (`int`, *optional*, defaults to `self.unet.config.sample_size * self.vae_scale_factor`):
              The height in pixels of the generated image.
          width (`int`, *optional*, defaults to `self.unet.config.sample_size * self.vae_scale_factor`):
              The width in pixels of the generated image.
          data_type (`int`, *optional*, defaults to 1):
              The data type (either 0 or 1). Only used if you are loading a checkpoint which supports a data type
              embedding; this is added for compatibility with the
              [UniDiffuser-v1](https://huggingface.co/thu-ml/unidiffuser-v1) checkpoint.
          num_inference_steps (`int`, *optional*, defaults to 50):
              The number of denoising steps. More denoising steps usually lead to a higher quality image at the
              expense of slower inference.
          guidance_scale (`float`, *optional*, defaults to 8.0):
              A higher guidance scale value encourages the model to generate images closely linked to the text
              `prompt` at the expense of lower image quality. Guidance scale is enabled when `guidance_scale > 1`.
          negative_prompt (`str` or `List[str]`, *optional*):
              The prompt or prompts to guide what to not include in image generation. If not defined, you need to
              pass `negative_prompt_embeds` instead. Ignored when not using guidance (`guidance_scale < 1`). Used in
              text-conditioned image generation (`text2img`) mode.
          num_images_per_prompt (`int`, *optional*, defaults to 1):
              The number of images to generate per prompt. Used in `text2img` (text-conditioned image generation) and
              `img` mode. If the mode is joint and both `num_images_per_prompt` and `num_prompts_per_image` are
              supplied, `min(num_images_per_prompt, num_prompts_per_image)` samples are generated.
          num_prompts_per_image (`int`, *optional*, defaults to 1):
              The number of prompts to generate per image. Used in `img2text` (image-conditioned text generation) and
              `text` mode. If the mode is joint and both `num_images_per_prompt` and `num_prompts_per_image` are
              supplied, `min(num_images_per_prompt, num_prompts_per_image)` samples are generated.
          eta (`float`, *optional*, defaults to 0.0):
              Corresponds to parameter eta (η) from the [DDIM](https://arxiv.org/abs/2010.02502) paper. Only applies
              to the [`~schedulers.DDIMScheduler`], and is ignored in other schedulers.
          generator (`torch.Generator` or `List[torch.Generator]`, *optional*):
              A [`torch.Generator`](https://pytorch.org/docs/stable/generated/torch.Generator.html) to make
              generation deterministic.
          latents (`torch.FloatTensor`, *optional*):
              Pre-generated noisy latents sampled from a Gaussian distribution, to be used as inputs for joint
              image-text generation. Can be used to tweak the same generation with different prompts. If not
              provided, a latents tensor is generated by sampling using the supplied random `generator`. This assumes
              a full set of VAE, CLIP, and text latents, if supplied, overrides the value of `prompt_latents`,
              `vae_latents`, and `clip_latents`.
          prompt_latents (`torch.FloatTensor`, *optional*):
              Pre-generated noisy latents sampled from a Gaussian distribution, to be used as inputs for text
              generation. Can be used to tweak the same generation with different prompts. If not provided, a latents
              tensor is generated by sampling using the supplied random `generator`.
          vae_latents (`torch.FloatTensor`, *optional*):
              Pre-generated noisy latents sampled from a Gaussian distribution, to be used as inputs for image
              generation. Can be used to tweak the same generation with different prompts. If not provided, a latents
              tensor is generated by sampling using the supplied random `generator`.
          clip_latents (`torch.FloatTensor`, *optional*):
              Pre-generated noisy latents sampled from a Gaussian distribution, to be used as inputs for image
              generation. Can be used to tweak the same generation with different prompts. If not provided, a latents
              tensor is generated by sampling using the supplied random `generator`.
          prompt_embeds (`torch.FloatTensor`, *optional*):
              Pre-generated text embeddings. Can be used to easily tweak text inputs (prompt weighting). If not
              provided, text embeddings are generated from the `prompt` input argument. Used in text-conditioned
              image generation (`text2img`) mode.
          negative_prompt_embeds (`torch.FloatTensor`, *optional*):
              Pre-generated negative text embeddings. Can be used to easily tweak text inputs (prompt weighting). If
              not provided, `negative_prompt_embeds` are be generated from the `negative_prompt` input argument. Used
              in text-conditioned image generation (`text2img`) mode.
          output_type (`str`, *optional*, defaults to `"pil"`):
              The output format of the generated image. Choose between `PIL.Image` or `np.array`.
          return_dict (`bool`, *optional*, defaults to `True`):
              Whether or not to return a [`~pipelines.ImageTextPipelineOutput`] instead of a plain tuple.
          callback (`Callable`, *optional*):
              A function that calls every `callback_steps` steps during inference. The function is called with the
              following arguments: `callback(step: int, timestep: int, latents: torch.FloatTensor)`.
          callback_steps (`int`, *optional*, defaults to 1):
              The frequency at which the `callback` function is called. If not specified, the callback is called at
              every step.
  
      Returns:
          [`~pipelines.unidiffuser.ImageTextPipelineOutput`] or `tuple`:
              If `return_dict` is `True`, [`~pipelines.unidiffuser.ImageTextPipelineOutput`] is returned, otherwise a
              `tuple` is returned where the first element is a list with the generated images and the second element
              is a list of generated texts.
      """
  
      # 0. Default height and width to unet
      height = height or self.unet_resolution * self.vae_scale_factor
      width = width or self.unet_resolution * self.vae_scale_factor
  
      # 1. Check inputs
      # Recalculate mode for each call to the pipeline.
      mode = self._infer_mode(
          prompt,
          prompt_embeds,
          image,
          latents,
          prompt_latents,
          vae_latents,
          clip_latents,
      )
      self.check_inputs(
          mode,
          prompt,
          image,
          height,
          width,
          callback_steps,
          negative_prompt,
          prompt_embeds,
          negative_prompt_embeds,
          latents,
          prompt_latents,
          vae_latents,
          clip_latents,
      )
  
      # 2. Define call parameters
      batch_size, multiplier = self._infer_batch_size(
          mode,
          prompt,
          prompt_embeds,
          image,
          num_images_per_prompt,
          num_prompts_per_image,
          latents,
          prompt_latents,
          vae_latents,
          clip_latents,
      )
      device = self._execution_device
      reduce_text_emb_dim = (
          self.text_intermediate_dim < self.text_encoder_hidden_size
          or self.mode != "text2img"
      )
  
      # here `guidance_scale` is defined analog to the guidance weight `w` of equation (2)
      # of the Imagen paper: https://arxiv.org/pdf/2205.11487.pdf . `guidance_scale = 1`
      # corresponds to doing no classifier free guidance.
      # Note that this differs from the formulation in the unidiffusers paper!
      do_classifier_free_guidance = guidance_scale > 1.0
  
      # check if scheduler is in sigmas space
      # scheduler_is_in_sigma_space = hasattr(self.scheduler, "sigmas")
  
      # 3. Encode input prompt, if available; otherwise prepare text latents
      if latents is not None:
          # Overwrite individual latents
          vae_latents, clip_latents, prompt_latents = self._split_joint(
              latents, height, width
          )
  
      if mode in ["text2img"]:
          # 3.1. Encode input prompt, if available
          assert prompt is not None or prompt_embeds is not None
          prompt_embeds, negative_prompt_embeds = self.encode_prompt(
              prompt=prompt,
              device=device,
              num_images_per_prompt=multiplier,
              do_classifier_free_guidance=do_classifier_free_guidance,
              negative_prompt=negative_prompt,
              prompt_embeds=prompt_embeds,
              negative_prompt_embeds=negative_prompt_embeds,
          )
  
          # if do_classifier_free_guidance:
          #     prompt_embeds = torch.cat([negative_prompt_embeds, prompt_embeds])
      else:
          # 3.2. Prepare text latent variables, if input not available
          prompt_embeds = self.prepare_text_latents(
              batch_size=batch_size,
              num_images_per_prompt=multiplier,
              seq_len=self.text_encoder_seq_len,
              hidden_size=self.text_encoder_hidden_size,
              dtype=self.text_encoder.dtype,  # Should work with both full precision and mixed precision
              device=device,
              generator=generator,
              latents=prompt_latents,
          )
  
      if reduce_text_emb_dim:
          prompt_embeds = self.text_decoder.encode(prompt_embeds)
  
      # 4. Encode image, if available; otherwise prepare image latents
      if mode in ["img2text"]:
          # 4.1. Encode images, if available
          assert (
              image is not None
          ), "`img2text` requires a conditioning image"
          # Encode image using VAE
          image_vae = self.image_processor.preprocess(image)
          height, width = image_vae.shape[-2:]
          image_vae_latents = self.encode_image_vae_latents(
              image=image_vae,
              batch_size=batch_size,
              num_prompts_per_image=multiplier,
              dtype=prompt_embeds.dtype,
              device=device,
              do_classifier_free_guidance=False,  # Copied from InstructPix2Pix, don't use their version of CFG
              generator=generator,
          )
  
          # Encode image using CLIP
          image_clip_latents = self.encode_image_clip_latents(
              image=image,
              batch_size=batch_size,
              num_prompts_per_image=multiplier,
              dtype=prompt_embeds.dtype,
              device=device,
              generator=generator,
          )
          # (batch_size, clip_hidden_size) => (batch_size, 1, clip_hidden_size)
          image_clip_latents = image_clip_latents.unsqueeze(1)
      else:
          # 4.2. Prepare image latent variables, if input not available
          # Prepare image VAE latents in latent space
          image_vae_latents = self.prepare_image_vae_latents(
              batch_size=batch_size,
              num_prompts_per_image=multiplier,
              num_channels_latents=self.num_channels_latents,
              height=height,
              width=width,
              dtype=prompt_embeds.dtype,
              device=device,
              generator=generator,
              latents=vae_latents,
          )
  
          # Prepare image CLIP latents
          image_clip_latents = self.prepare_image_clip_latents(
              batch_size=batch_size,
              num_prompts_per_image=multiplier,
              clip_img_dim=self.image_encoder_projection_dim,
              dtype=prompt_embeds.dtype,
              device=device,
              generator=generator,
              latents=clip_latents,
          )
  
      # 5. Set timesteps
      self.scheduler.set_timesteps(num_inference_steps, device=device)
      timesteps = self.scheduler.timesteps
      # max_timestep = timesteps[0]
      max_timestep = self.scheduler.config.num_train_timesteps
  
      # 6. Prepare latent variables
      if mode == "joint":
          latents = self._combine_joint(
              image_vae_latents, image_clip_latents, prompt_embeds
          )
      elif mode in ["text2img", "img"]:
          latents = self._combine(image_vae_latents, image_clip_latents)
      elif mode in ["img2text", "text"]:
          latents = prompt_embeds
  
      # 7. Prepare extra step kwargs. TODO: Logic should ideally just be moved out of the pipeline
      extra_step_kwargs = self.prepare_extra_step_kwargs(generator, eta)
  
      logger.debug(f"Scheduler extra step kwargs: {extra_step_kwargs}")
  
      # 8. Denoising loop
      num_warmup_steps = (
          len(timesteps) - num_inference_steps * self.scheduler.order
      )
      with self.progress_bar(total=num_inference_steps) as progress_bar:
          for i, t in enumerate(timesteps):
              # predict the noise residual
              # Also applies classifier-free guidance as described in the UniDiffuser paper
              noise_pred = self._get_noise_pred(
                  mode,
                  latents,
                  t,
                  prompt_embeds,
                  image_vae_latents,
                  image_clip_latents,
                  max_timestep,
                  data_type,
                  guidance_scale,
                  generator,
                  device,
                  height,
                  width,
              )
  
              # compute the previous noisy sample x_t -> x_t-1
              latents = self.scheduler.step(
                  noise_pred, t, latents, **extra_step_kwargs
              ).prev_sample
  
              # call the callback, if provided
              if i == len(timesteps) - 1 or (
                  (i + 1) > num_warmup_steps
                  and (i + 1) % self.scheduler.order == 0
              ):
                  progress_bar.update()
                  if callback is not None and i % callback_steps == 0:
                      step_idx = i // getattr(
                          self.scheduler, "order", 1
                      )
                      callback(step_idx, t, latents)
  
      # 9. Post-processing
      image = None
      text = None
      if mode == "joint":
          (
              image_vae_latents,
              image_clip_latents,
              text_latents,
          ) = self._split_joint(latents, height, width)
  
          if not output_type == "latent":
              # Map latent VAE image back to pixel space
              image = self.vae.decode(
                  image_vae_latents / self.vae.config.scaling_factor,
                  return_dict=False,
              )[0]
          else:
              image = image_vae_latents
  
          text = self.decode_text_latents(text_latents, device)
      elif mode in ["text2img", "img"]:
          image_vae_latents, image_clip_latents = self._split(
              latents, height, width
          )
  
          if not output_type == "latent":
              # Map latent VAE image back to pixel space
              image = self.vae.decode(
                  image_vae_latents / self.vae.config.scaling_factor,
                  return_dict=False,
              )[0]
          else:
              image = image_vae_latents
      elif mode in ["img2text", "text"]:
          text_latents = latents
          text = self.decode_text_latents(text_latents, device)
  
      self.maybe_free_model_hooks()
  
      # 10. Postprocess the image, if necessary
      if image is not None:
          do_denormalize = [True] * image.shape[0]
          image = self.image_processor.postprocess(
              image,
              output_type=output_type,
              do_denormalize=do_denormalize,
          )
  
      # Offload last model to CPU
      if (
          hasattr(self, "final_offload_hook")
          and self.final_offload_hook is not None
      ):
          self.final_offload_hook.offload()
  
      if not return_dict:
          return (image, text)
  
      return ImageTextPipelineOutput(images=image, text=text)
  
imports: import inspect
  from dataclasses import dataclass
  from typing import Callable, List, Optional, Union
  import numpy
  import PIL.Image
  import torch
  from transformers import CLIPImageProcessor, CLIPTextModel, CLIPTokenizer, CLIPVisionModelWithProjection, GPT2Tokenizer
  from ...image_processor import VaeImageProcessor
  from ...loaders import LoraLoaderMixin, TextualInversionLoaderMixin
  from ...models import AutoencoderKL
  from ...models.lora import adjust_lora_scale_text_encoder
  from ...schedulers import KarrasDiffusionSchedulers
  from ...utils import USE_PEFT_BACKEND, deprecate, logging, scale_lora_layers, unscale_lora_layers
  from ...utils.outputs import BaseOutput
  from ...utils.torch_utils import randn_tensor
  from ..pipeline_utils import DiffusionPipeline
  from .modeling_text_decoder import UniDiffuserTextDecoder
  from .modeling_uvit import UniDiffuserModel
label_desc: Use image node_id 30 for calling the UniDiffuserPipeline obj's underlying member method code with example usage: obj.__call__(prompt, image, height, width, data_type, num_inference_steps, guidance_scale, negative_prompt, num_images_per_prompt, num_prompts_per_image, eta, generator, latents, prompt_latents, vae_latents, clip_latents, prompt_embeds, negative_prompt_embeds, output_type, return_dict, callback, callback_steps) and returns: ImageTextPipelineOutput, image,  text