class_docstr: Pipeline for upscaling Stable Diffusion output image resolution by a factor of 2.
  
  This model inherits from [`DiffusionPipeline`]. Check the superclass documentation for the generic methods
  implemented for all pipelines (downloading, saving, running on a particular device, etc.).
  
  Args:
      vae ([`AutoencoderKL`]):
          Variational Auto-Encoder (VAE) model to encode and decode images to and from latent representations.
      text_encoder ([`~transformers.CLIPTextModel`]):
          Frozen text-encoder ([clip-vit-large-patch14](https://huggingface.co/openai/clip-vit-large-patch14)).
      tokenizer ([`~transformers.CLIPTokenizer`]):
          A `CLIPTokenizer` to tokenize text.
      unet ([`UNet2DConditionModel`]):
          A `UNet2DConditionModel` to denoise the encoded image latents.
      scheduler ([`SchedulerMixin`]):
          A [`EulerDiscreteScheduler`] to be used in combination with `unet` to denoise the encoded image latents.
class_docstr_tok: ["Pipeline","for","upscaling","Stable","Diffusion","output","image","resolution","by","a","factor","of","2",".","This","model","inherits","from","[","`","DiffusionPipeline","`","]",".","Check","the","superclass","documentation","for","the","generic","methods","implemented","for","all","pipelines","(","downloading",",","saving",",","running","on","a","particular","device",",","etc.",")",".","Args",":","vae","(","[","`","AutoencoderKL","`","]",")",":","Variational","Auto-Encoder","(","VAE",")","model","to","encode","and","decode","images","to","and","from","latent","representations",".","text_encoder","(","[","`","~transformers.CLIPTextModel","`","]",")",":","Frozen","text-encoder","(","[","clip-vit-large-patch14","]","(","https",":","\/\/huggingface.co\/openai\/clip-vit-large-patch14",")",")",".","tokenizer","(","[","`","~transformers.CLIPTokenizer","`","]",")",":","A","`","CLIPTokenizer","`","to","tokenize","text",".","unet","(","[","`","UNet2DConditionModel","`","]",")",":","A","`","UNet2DConditionModel","`","to","denoise","the","encoded","image","latents",".","scheduler","(","[","`","SchedulerMixin","`","]",")",":","A","[","`","EulerDiscreteScheduler","`","]","to","be","used","in","combination","with","`","unet","`","to","denoise","the","encoded","image","latents","."]
docstr: The call function to the pipeline for generation.
  
  Args:
      prompt (`str` or `List[str]`):
          The prompt or prompts to guide image upscaling.
      image (`torch.FloatTensor`, `PIL.Image.Image`, `np.ndarray`, `List[torch.FloatTensor]`, `List[PIL.Image.Image]`, or `List[np.ndarray]`):
          `Image` or tensor representing an image batch to be upscaled. If it's a tensor, it can be either a
          latent output from a Stable Diffusion model or an image tensor in the range `[-1, 1]`. It is considered
          a `latent` if `image.shape[1]` is `4`; otherwise, it is considered to be an image representation and
          encoded using this pipeline's `vae` encoder.
      num_inference_steps (`int`, *optional*, defaults to 50):
          The number of denoising steps. More denoising steps usually lead to a higher quality image at the
          expense of slower inference.
      guidance_scale (`float`, *optional*, defaults to 7.5):
          A higher guidance scale value encourages the model to generate images closely linked to the text
          `prompt` at the expense of lower image quality. Guidance scale is enabled when `guidance_scale > 1`.
      negative_prompt (`str` or `List[str]`, *optional*):
          The prompt or prompts to guide what to not include in image generation. If not defined, you need to
          pass `negative_prompt_embeds` instead. Ignored when not using guidance (`guidance_scale < 1`).
      eta (`float`, *optional*, defaults to 0.0):
          Corresponds to parameter eta (η) from the [DDIM](https://arxiv.org/abs/2010.02502) paper. Only applies
          to the [`~schedulers.DDIMScheduler`], and is ignored in other schedulers.
      generator (`torch.Generator` or `List[torch.Generator]`, *optional*):
          A [`torch.Generator`](https://pytorch.org/docs/stable/generated/torch.Generator.html) to make
          generation deterministic.
      latents (`torch.FloatTensor`, *optional*):
          Pre-generated noisy latents sampled from a Gaussian distribution, to be used as inputs for image
          generation. Can be used to tweak the same generation with different prompts. If not provided, a latents
          tensor is generated by sampling using the supplied random `generator`.
      output_type (`str`, *optional*, defaults to `"pil"`):
          The output format of the generated image. Choose between `PIL.Image` or `np.array`.
      return_dict (`bool`, *optional*, defaults to `True`):
          Whether or not to return a [`~pipelines.stable_diffusion.StableDiffusionPipelineOutput`] instead of a
          plain tuple.
      callback (`Callable`, *optional*):
          A function that calls every `callback_steps` steps during inference. The function is called with the
          following arguments: `callback(step: int, timestep: int, latents: torch.FloatTensor)`.
      callback_steps (`int`, *optional*, defaults to 1):
          The frequency at which the `callback` function is called. If not specified, the callback is called at
          every step.
  
  Examples:
  ```py
  >>> from diffusers import StableDiffusionLatentUpscalePipeline, StableDiffusionPipeline
  >>> import torch
  
  
  >>> pipeline = StableDiffusionPipeline.from_pretrained(
  ...     "CompVis/stable-diffusion-v1-4", torch_dtype=torch.float16
  ... )
  >>> pipeline.to("cuda")
  
  >>> model_id = "stabilityai/sd-x2-latent-upscaler"
  >>> upscaler = StableDiffusionLatentUpscalePipeline.from_pretrained(model_id, torch_dtype=torch.float16)
  >>> upscaler.to("cuda")
  
  >>> prompt = "a photo of an astronaut high resolution, unreal engine, ultra realistic"
  >>> generator = torch.manual_seed(33)
  
  >>> low_res_latents = pipeline(prompt, generator=generator, output_type="latent").images
  
  >>> with torch.no_grad():
  ...     image = pipeline.decode_latents(low_res_latents)
  >>> image = pipeline.numpy_to_pil(image)[0]
  
  >>> image.save("../images/a1.png")
  
  >>> upscaled_image = upscaler(
  ...     prompt=prompt,
  ...     image=low_res_latents,
  ...     num_inference_steps=20,
  ...     guidance_scale=0,
  ...     generator=generator,
  ... ).images[0]
  
  >>> upscaled_image.save("../images/a2.png")
  ```
  
  Returns:
      [`~pipelines.stable_diffusion.StableDiffusionPipelineOutput`] or `tuple`:
          If `return_dict` is `True`, [`~pipelines.stable_diffusion.StableDiffusionPipelineOutput`] is returned,
          otherwise a `tuple` is returned where the first element is a list with the generated images.
docstr_tok: ["The","call","function","to","the","pipeline","for","generation",".","Args",":","prompt","(","`","str","`","or","`","List","[","str","]","`",")",":","The","prompt","or","prompts","to","guide","image","upscaling",".","image","(","`","torch.FloatTensor","`",",","`","PIL.Image.Image","`",",","`","np.ndarray","`",",","`","List","[","torch.FloatTensor","]","`",",","`","List","[","PIL.Image.Image","]","`",",","or","`","List","[","np.ndarray","]","`",")",":","`","Image","`","or","tensor","representing","an","image","batch","to","be","upscaled",".","If","it","'s","a","tensor",",","it","can","be","either","a","latent","output","from","a","Stable","Diffusion","model","or","an","image","tensor","in","the","range","`","[","-1",",","1","]","`",".","It","is","considered","a","`","latent","`","if","`","image.shape","[","1","]","`","is","`","4","`",";","otherwise",",","it","is","considered","to","be","an","image","representation","and","encoded","using","this","pipeline","'s","`","vae","`","encoder",".","num_inference_steps","(","`","int","`",",","*","optional","*",",","defaults","to","50",")",":","The","number","of","denoising","steps",".","More","denoising","steps","usually","lead","to","a","higher","quality","image","at","the","expense","of","slower","inference",".","guidance_scale","(","`","float","`",",","*","optional","*",",","defaults","to","7.5",")",":","A","higher","guidance","scale","value","encourages","the","model","to","generate","images","closely","linked","to","the","text","`","prompt","`","at","the","expense","of","lower","image","quality",".","Guidance","scale","is","enabled","when","`","guidance_scale",">","1","`",".","negative_prompt","(","`","str","`","or","`","List","[","str","]","`",",","*","optional","*",")",":","The","prompt","or","prompts","to","guide","what","to","not","include","in","image","generation",".","If","not","defined",",","you","need","to","pass","`","negative_prompt_embeds","`","instead",".","Ignored","when","not","using","guidance","(","`","guidance_scale","<","1","`",")",".","eta","(","`","float","`",",","*","optional","*",",","defaults","to","0.0",")",":","Corresponds","to","parameter","eta","(","\u03b7",")","from","the","[","DDIM","]","(","https",":","\/\/arxiv.org\/abs\/2010.02502",")","paper",".","Only","applies","to","the","[","`","~schedulers.DDIMScheduler","`","]",",","and","is","ignored","in","other","schedulers",".","generator","(","`","torch.Generator","`","or","`","List","[","torch.Generator","]","`",",","*","optional","*",")",":","A","[","`","torch.Generator","`","]","(","https",":","\/\/pytorch.org\/docs\/stable\/generated\/torch.Generator.html",")","to","make","generation","deterministic",".","latents","(","`","torch.FloatTensor","`",",","*","optional","*",")",":","Pre-generated","noisy","latents","sampled","from","a","Gaussian","distribution",",","to","be","used","as","inputs","for","image","generation",".","Can","be","used","to","tweak","the","same","generation","with","different","prompts",".","If","not","provided",",","a","latents","tensor","is","generated","by","sampling","using","the","supplied","random","`","generator","`",".","output_type","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","pil","''","`",")",":","The","output","format","of","the","generated","image",".","Choose","between","`","PIL.Image","`","or","`","np.array","`",".","return_dict","(","`","bool","`",",","*","optional","*",",","defaults","to","`","True","`",")",":","Whether","or","not","to","return","a","[","`","~pipelines.stable_diffusion.StableDiffusionPipelineOutput","`","]","instead","of","a","plain","tuple",".","callback","(","`","Callable","`",",","*","optional","*",")",":","A","function","that","calls","every","`","callback_steps","`","steps","during","inference",".","The","function","is","called","with","the","following","arguments",":","`","callback","(","step",":","int",",","timestep",":","int",",","latents",":","torch.FloatTensor",")","`",".","callback_steps","(","`","int","`",",","*","optional","*",",","defaults","to","1",")",":","The","frequency","at","which","the","`","callback","`","function","is","called",".","If","not","specified",",","the","callback","is","called","at","every","step",".","Examples",":","``","`","py",">",">",">","from","diffusers","import","StableDiffusionLatentUpscalePipeline",",","StableDiffusionPipeline",">",">",">","import","torch",">",">",">","pipeline","=","StableDiffusionPipeline.from_pretrained","(","...","``","CompVis\/stable-diffusion-v1-4","''",",","torch_dtype=torch.float16","...",")",">",">",">","pipeline.to","(","``","cuda","''",")",">",">",">","model_id","=","``","stabilityai\/sd-x2-latent-upscaler","''",">",">",">","upscaler","=","StableDiffusionLatentUpscalePipeline.from_pretrained","(","model_id",",","torch_dtype=torch.float16",")",">",">",">","upscaler.to","(","``","cuda","''",")",">",">",">","prompt","=","``","a","photo","of","an","astronaut","high","resolution",",","unreal","engine",",","ultra","realistic","''",">",">",">","generator","=","torch.manual_seed","(","33",")",">",">",">","low_res_latents","=","pipeline","(","prompt",",","generator=generator",",","output_type=","''","latent","''",")",".images",">",">",">","with","torch.no_grad","(",")",":","...","image","=","pipeline.decode_latents","(","low_res_latents",")",">",">",">","image","=","pipeline.numpy_to_pil","(","image",")","[","0","]",">",">",">","image.save","(","``","..","\/images\/a1.png","''",")",">",">",">","upscaled_image","=","upscaler","(","...","prompt=prompt",",","...","image=low_res_latents",",","...","num_inference_steps=20",",","...","guidance_scale=0",",","...","generator=generator",",","...",")",".images","[","0","]",">",">",">","upscaled_image.save","(","``","..","\/images\/a2.png","''",")","``","`","Returns",":","[","`","~pipelines.stable_diffusion.StableDiffusionPipelineOutput","`","]","or","`","tuple","`",":","If","`","return_dict","`","is","`","True","`",",","[","`","~pipelines.stable_diffusion.StableDiffusionPipelineOutput","`","]","is","returned",",","otherwise","a","`","tuple","`","is","returned","where","the","first","element","is","a","list","with","the","generated","images","."]
code: def __call__(
      self,
      prompt: Union[str, List[str]],
      image: PipelineImageInput = None,
      num_inference_steps: int = 75,
      guidance_scale: float = 9.0,
      negative_prompt: Optional[Union[str, List[str]]] = None,
      generator: Optional[
          Union[torch.Generator, List[torch.Generator]]
      ] = None,
      latents: Optional[torch.FloatTensor] = None,
      output_type: Optional[str] = "pil",
      return_dict: bool = True,
      callback: Optional[
          Callable[[int, int, torch.FloatTensor], None]
      ] = None,
      callback_steps: int = 1,
  ):
      r"""
      The call function to the pipeline for generation.
  
      Args:
          prompt (`str` or `List[str]`):
              The prompt or prompts to guide image upscaling.
          image (`torch.FloatTensor`, `PIL.Image.Image`, `np.ndarray`, `List[torch.FloatTensor]`, `List[PIL.Image.Image]`, or `List[np.ndarray]`):
              `Image` or tensor representing an image batch to be upscaled. If it's a tensor, it can be either a
              latent output from a Stable Diffusion model or an image tensor in the range `[-1, 1]`. It is considered
              a `latent` if `image.shape[1]` is `4`; otherwise, it is considered to be an image representation and
              encoded using this pipeline's `vae` encoder.
          num_inference_steps (`int`, *optional*, defaults to 50):
              The number of denoising steps. More denoising steps usually lead to a higher quality image at the
              expense of slower inference.
          guidance_scale (`float`, *optional*, defaults to 7.5):
              A higher guidance scale value encourages the model to generate images closely linked to the text
              `prompt` at the expense of lower image quality. Guidance scale is enabled when `guidance_scale > 1`.
          negative_prompt (`str` or `List[str]`, *optional*):
              The prompt or prompts to guide what to not include in image generation. If not defined, you need to
              pass `negative_prompt_embeds` instead. Ignored when not using guidance (`guidance_scale < 1`).
          eta (`float`, *optional*, defaults to 0.0):
              Corresponds to parameter eta (η) from the [DDIM](https://arxiv.org/abs/2010.02502) paper. Only applies
              to the [`~schedulers.DDIMScheduler`], and is ignored in other schedulers.
          generator (`torch.Generator` or `List[torch.Generator]`, *optional*):
              A [`torch.Generator`](https://pytorch.org/docs/stable/generated/torch.Generator.html) to make
              generation deterministic.
          latents (`torch.FloatTensor`, *optional*):
              Pre-generated noisy latents sampled from a Gaussian distribution, to be used as inputs for image
              generation. Can be used to tweak the same generation with different prompts. If not provided, a latents
              tensor is generated by sampling using the supplied random `generator`.
          output_type (`str`, *optional*, defaults to `"pil"`):
              The output format of the generated image. Choose between `PIL.Image` or `np.array`.
          return_dict (`bool`, *optional*, defaults to `True`):
              Whether or not to return a [`~pipelines.stable_diffusion.StableDiffusionPipelineOutput`] instead of a
              plain tuple.
          callback (`Callable`, *optional*):
              A function that calls every `callback_steps` steps during inference. The function is called with the
              following arguments: `callback(step: int, timestep: int, latents: torch.FloatTensor)`.
          callback_steps (`int`, *optional*, defaults to 1):
              The frequency at which the `callback` function is called. If not specified, the callback is called at
              every step.
  
      Examples:
      ```py
      >>> from diffusers import StableDiffusionLatentUpscalePipeline, StableDiffusionPipeline
      >>> import torch
  
  
      >>> pipeline = StableDiffusionPipeline.from_pretrained(
      ...     "CompVis/stable-diffusion-v1-4", torch_dtype=torch.float16
      ... )
      >>> pipeline.to("cuda")
  
      >>> model_id = "stabilityai/sd-x2-latent-upscaler"
      >>> upscaler = StableDiffusionLatentUpscalePipeline.from_pretrained(model_id, torch_dtype=torch.float16)
      >>> upscaler.to("cuda")
  
      >>> prompt = "a photo of an astronaut high resolution, unreal engine, ultra realistic"
      >>> generator = torch.manual_seed(33)
  
      >>> low_res_latents = pipeline(prompt, generator=generator, output_type="latent").images
  
      >>> with torch.no_grad():
      ...     image = pipeline.decode_latents(low_res_latents)
      >>> image = pipeline.numpy_to_pil(image)[0]
  
      >>> image.save("../images/a1.png")
  
      >>> upscaled_image = upscaler(
      ...     prompt=prompt,
      ...     image=low_res_latents,
      ...     num_inference_steps=20,
      ...     guidance_scale=0,
      ...     generator=generator,
      ... ).images[0]
  
      >>> upscaled_image.save("../images/a2.png")
      ```
  
      Returns:
          [`~pipelines.stable_diffusion.StableDiffusionPipelineOutput`] or `tuple`:
              If `return_dict` is `True`, [`~pipelines.stable_diffusion.StableDiffusionPipelineOutput`] is returned,
              otherwise a `tuple` is returned where the first element is a list with the generated images.
      """
  
      # 1. Check inputs
      self.check_inputs(prompt, image, callback_steps)
  
      # 2. Define call parameters
      batch_size = 1 if isinstance(prompt, str) else len(prompt)
      device = self._execution_device
      # here `guidance_scale` is defined analog to the guidance weight `w` of equation (2)
      # of the Imagen paper: https://arxiv.org/pdf/2205.11487.pdf . `guidance_scale = 1`
      # corresponds to doing no classifier free guidance.
      do_classifier_free_guidance = guidance_scale > 1.0
  
      if guidance_scale == 0:
          prompt = [""] * batch_size
  
      # 3. Encode input prompt
      text_embeddings, text_pooler_out = self._encode_prompt(
          prompt, device, do_classifier_free_guidance, negative_prompt
      )
  
      # 4. Preprocess image
      image = self.image_processor.preprocess(image)
      image = image.to(dtype=text_embeddings.dtype, device=device)
      if image.shape[1] == 3:
          # encode image if not in latent-space yet
          image = (
              self.vae.encode(image).latent_dist.sample()
              * self.vae.config.scaling_factor
          )
  
      # 5. set timesteps
      self.scheduler.set_timesteps(num_inference_steps, device=device)
      timesteps = self.scheduler.timesteps
  
      batch_multiplier = 2 if do_classifier_free_guidance else 1
      image = image[None, :] if image.ndim == 3 else image
      image = torch.cat([image] * batch_multiplier)
  
      # 5. Add noise to image (set to be 0):
      # (see below notes from the author):
      # "the This step theoretically can make the model work better on out-of-distribution inputs, but mostly just seems to make it match the input less, so it's turned off by default."
      noise_level = torch.tensor(
          [0.0], dtype=torch.float32, device=device
      )
      noise_level = torch.cat([noise_level] * image.shape[0])
      inv_noise_level = (noise_level**2 + 1) ** (-0.5)
  
      image_cond = (
          F.interpolate(image, scale_factor=2, mode="nearest")
          * inv_noise_level[:, None, None, None]
      )
      image_cond = image_cond.to(text_embeddings.dtype)
  
      noise_level_embed = torch.cat(
          [
              torch.ones(
                  text_pooler_out.shape[0],
                  64,
                  dtype=text_pooler_out.dtype,
                  device=device,
              ),
              torch.zeros(
                  text_pooler_out.shape[0],
                  64,
                  dtype=text_pooler_out.dtype,
                  device=device,
              ),
          ],
          dim=1,
      )
  
      timestep_condition = torch.cat(
          [noise_level_embed, text_pooler_out], dim=1
      )
  
      # 6. Prepare latent variables
      height, width = image.shape[2:]
      num_channels_latents = self.vae.config.latent_channels
      latents = self.prepare_latents(
          batch_size,
          num_channels_latents,
          height * 2,  # 2x upscale
          width * 2,
          text_embeddings.dtype,
          device,
          generator,
          latents,
      )
  
      # 7. Check that sizes of image and latents match
      num_channels_image = image.shape[1]
      if (
          num_channels_latents + num_channels_image
          != self.unet.config.in_channels
      ):
          raise ValueError(
              f"Incorrect configuration settings! The config of `pipeline.unet`: {self.unet.config} expects"
              f" {self.unet.config.in_channels} but received `num_channels_latents`: {num_channels_latents} +"
              f" `num_channels_image`: {num_channels_image} "
              f" = {num_channels_latents+num_channels_image}. Please verify the config of"
              " `pipeline.unet` or your `image` input."
          )
  
      # 9. Denoising loop
      num_warmup_steps = 0
  
      with self.progress_bar(total=num_inference_steps) as progress_bar:
          for i, t in enumerate(timesteps):
              sigma = self.scheduler.sigmas[i]
              # expand the latents if we are doing classifier free guidance
              latent_model_input = (
                  torch.cat([latents] * 2)
                  if do_classifier_free_guidance
                  else latents
              )
              scaled_model_input = self.scheduler.scale_model_input(
                  latent_model_input, t
              )
  
              scaled_model_input = torch.cat(
                  [scaled_model_input, image_cond], dim=1
              )
              # preconditioning parameter based on  Karras et al. (2022) (table 1)
              timestep = torch.log(sigma) * 0.25
  
              noise_pred = self.unet(
                  scaled_model_input,
                  timestep,
                  encoder_hidden_states=text_embeddings,
                  timestep_cond=timestep_condition,
              ).sample
  
              # in original repo, the output contains a variance channel that's not used
              noise_pred = noise_pred[:, :-1]
  
              # apply preconditioning, based on table 1 in Karras et al. (2022)
              inv_sigma = 1 / (sigma**2 + 1)
              noise_pred = (
                  inv_sigma * latent_model_input
                  + self.scheduler.scale_model_input(sigma, t)
                  * noise_pred
              )
  
              # perform guidance
              if do_classifier_free_guidance:
                  noise_pred_uncond, noise_pred_text = noise_pred.chunk(
                      2
                  )
                  noise_pred = noise_pred_uncond + guidance_scale * (
                      noise_pred_text - noise_pred_uncond
                  )
  
              # compute the previous noisy sample x_t -> x_t-1
              latents = self.scheduler.step(
                  noise_pred, t, latents
              ).prev_sample
  
              # call the callback, if provided
              if i == len(timesteps) - 1 or (
                  (i + 1) > num_warmup_steps
                  and (i + 1) % self.scheduler.order == 0
              ):
                  progress_bar.update()
                  if callback is not None and i % callback_steps == 0:
                      step_idx = i // getattr(
                          self.scheduler, "order", 1
                      )
                      callback(step_idx, t, latents)
  
      if not output_type == "latent":
          image = self.vae.decode(
              latents / self.vae.config.scaling_factor,
              return_dict=False,
          )[0]
      else:
          image = latents
  
      image = self.image_processor.postprocess(
          image, output_type=output_type
      )
  
      if not return_dict:
          return (image,)
  
      return ImagePipelineOutput(images=image)
  
imports: import warnings
  from typing import Callable, List, Optional, Union
  import numpy
  import PIL.Image
  import torch
  import torch.nn.functional
  from transformers import CLIPTextModel, CLIPTokenizer
  from ...image_processor import PipelineImageInput, VaeImageProcessor
  from ...loaders import FromSingleFileMixin
  from ...models import AutoencoderKL, UNet2DConditionModel
  from ...schedulers import EulerDiscreteScheduler
  from ...utils import deprecate, logging
  from ...utils.torch_utils import randn_tensor
  from ..pipeline_utils import DiffusionPipeline, ImagePipelineOutput
label_desc: Use image node_id 8 for calling the StableDiffusionLatentUpscalePipeline obj's underlying member method code with example usage: obj.__call__(prompt, image, num_inference_steps, guidance_scale, negative_prompt, generator, latents, output_type, return_dict, callback, callback_steps) and returns: ImagePipelineOutput, image