class_docstr: Pipeline for text-to-video generation.
  
  This model inherits from [`DiffusionPipeline`]. Check the superclass documentation for the generic methods
  implemented for all pipelines (downloading, saving, running on a particular device, etc.).
  
  The pipeline also inherits the following loading methods:
      - [`~loaders.TextualInversionLoaderMixin.load_textual_inversion`] for loading textual inversion embeddings
      - [`~loaders.LoraLoaderMixin.load_lora_weights`] for loading LoRA weights
      - [`~loaders.LoraLoaderMixin.save_lora_weights`] for saving LoRA weights
      - [`~loaders.IPAdapterMixin.load_ip_adapter`] for loading IP Adapters
  
  Args:
      vae ([`AutoencoderKL`]):
          Variational Auto-Encoder (VAE) Model to encode and decode images to and from latent representations.
      text_encoder ([`CLIPTextModel`]):
          Frozen text-encoder ([clip-vit-large-patch14](https://huggingface.co/openai/clip-vit-large-patch14)).
      tokenizer (`CLIPTokenizer`):
          A [`~transformers.CLIPTokenizer`] to tokenize text.
      unet ([`UNet2DConditionModel`]):
          A [`UNet2DConditionModel`] used to create a UNetMotionModel to denoise the encoded video latents.
      motion_adapter ([`MotionAdapter`]):
          A [`MotionAdapter`] to be used in combination with `unet` to denoise the encoded video latents.
      scheduler ([`SchedulerMixin`]):
          A scheduler to be used in combination with `unet` to denoise the encoded image latents. Can be one of
          [`DDIMScheduler`], [`LMSDiscreteScheduler`], or [`PNDMScheduler`].
class_docstr_tok: ["Pipeline","for","text-to-video","generation",".","This","model","inherits","from","[","`","DiffusionPipeline","`","]",".","Check","the","superclass","documentation","for","the","generic","methods","implemented","for","all","pipelines","(","downloading",",","saving",",","running","on","a","particular","device",",","etc.",")",".","The","pipeline","also","inherits","the","following","loading","methods",":","-","[","`","~loaders.TextualInversionLoaderMixin.load_textual_inversion","`","]","for","loading","textual","inversion","embeddings","-","[","`","~loaders.LoraLoaderMixin.load_lora_weights","`","]","for","loading","LoRA","weights","-","[","`","~loaders.LoraLoaderMixin.save_lora_weights","`","]","for","saving","LoRA","weights","-","[","`","~loaders.IPAdapterMixin.load_ip_adapter","`","]","for","loading","IP","Adapters","Args",":","vae","(","[","`","AutoencoderKL","`","]",")",":","Variational","Auto-Encoder","(","VAE",")","Model","to","encode","and","decode","images","to","and","from","latent","representations",".","text_encoder","(","[","`","CLIPTextModel","`","]",")",":","Frozen","text-encoder","(","[","clip-vit-large-patch14","]","(","https",":","\/\/huggingface.co\/openai\/clip-vit-large-patch14",")",")",".","tokenizer","(","`","CLIPTokenizer","`",")",":","A","[","`","~transformers.CLIPTokenizer","`","]","to","tokenize","text",".","unet","(","[","`","UNet2DConditionModel","`","]",")",":","A","[","`","UNet2DConditionModel","`","]","used","to","create","a","UNetMotionModel","to","denoise","the","encoded","video","latents",".","motion_adapter","(","[","`","MotionAdapter","`","]",")",":","A","[","`","MotionAdapter","`","]","to","be","used","in","combination","with","`","unet","`","to","denoise","the","encoded","video","latents",".","scheduler","(","[","`","SchedulerMixin","`","]",")",":","A","scheduler","to","be","used","in","combination","with","`","unet","`","to","denoise","the","encoded","image","latents",".","Can","be","one","of","[","`","DDIMScheduler","`","]",",","[","`","LMSDiscreteScheduler","`","]",",","or","[","`","PNDMScheduler","`","]","."]
docstr: The call function to the pipeline for generation.
  
  Args:
      prompt (`str` or `List[str]`, *optional*):
          The prompt or prompts to guide image generation. If not defined, you need to pass `prompt_embeds`.
      height (`int`, *optional*, defaults to `self.unet.config.sample_size * self.vae_scale_factor`):
          The height in pixels of the generated video.
      width (`int`, *optional*, defaults to `self.unet.config.sample_size * self.vae_scale_factor`):
          The width in pixels of the generated video.
      num_frames (`int`, *optional*, defaults to 16):
          The number of video frames that are generated. Defaults to 16 frames which at 8 frames per seconds
          amounts to 2 seconds of video.
      num_inference_steps (`int`, *optional*, defaults to 50):
          The number of denoising steps. More denoising steps usually lead to a higher quality videos at the
          expense of slower inference.
      guidance_scale (`float`, *optional*, defaults to 7.5):
          A higher guidance scale value encourages the model to generate images closely linked to the text
          `prompt` at the expense of lower image quality. Guidance scale is enabled when `guidance_scale > 1`.
      negative_prompt (`str` or `List[str]`, *optional*):
          The prompt or prompts to guide what to not include in image generation. If not defined, you need to
          pass `negative_prompt_embeds` instead. Ignored when not using guidance (`guidance_scale < 1`).
      eta (`float`, *optional*, defaults to 0.0):
          Corresponds to parameter eta (η) from the [DDIM](https://arxiv.org/abs/2010.02502) paper. Only applies
          to the [`~schedulers.DDIMScheduler`], and is ignored in other schedulers.
      generator (`torch.Generator` or `List[torch.Generator]`, *optional*):
          A [`torch.Generator`](https://pytorch.org/docs/stable/generated/torch.Generator.html) to make
          generation deterministic.
      latents (`torch.FloatTensor`, *optional*):
          Pre-generated noisy latents sampled from a Gaussian distribution, to be used as inputs for video
          generation. Can be used to tweak the same generation with different prompts. If not provided, a latents
          tensor is generated by sampling using the supplied random `generator`. Latents should be of shape
          `(batch_size, num_channel, num_frames, height, width)`.
      prompt_embeds (`torch.FloatTensor`, *optional*):
          Pre-generated text embeddings. Can be used to easily tweak text inputs (prompt weighting). If not
          provided, text embeddings are generated from the `prompt` input argument.
      negative_prompt_embeds (`torch.FloatTensor`, *optional*):
          Pre-generated negative text embeddings. Can be used to easily tweak text inputs (prompt weighting). If
          not provided, `negative_prompt_embeds` are generated from the `negative_prompt` input argument.
      ip_adapter_image (`PipelineImageInput`, *optional*):
          Optional image input to work with IP Adapters.
      conditioning_frames (`List[PipelineImageInput]`, *optional*):
          The ControlNet input condition to provide guidance to the `unet` for generation. If multiple ControlNets
          are specified, images must be passed as a list such that each element of the list can be correctly
          batched for input to a single ControlNet.
      output_type (`str`, *optional*, defaults to `"pil"`):
          The output format of the generated video. Choose between `torch.FloatTensor`, `PIL.Image` or
          `np.array`.
      return_dict (`bool`, *optional*, defaults to `True`):
          Whether or not to return a [`~pipelines.text_to_video_synthesis.TextToVideoSDPipelineOutput`] instead
          of a plain tuple.
      cross_attention_kwargs (`dict`, *optional*):
          A kwargs dictionary that if specified is passed along to the [`AttentionProcessor`] as defined in
          [`self.processor`](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/attention_processor.py).
      controlnet_conditioning_scale (`float` or `List[float]`, *optional*, defaults to 1.0):
          The outputs of the ControlNet are multiplied by `controlnet_conditioning_scale` before they are added
          to the residual in the original `unet`. If multiple ControlNets are specified in `init`, you can set
          the corresponding scale as a list.
      guess_mode (`bool`, *optional*, defaults to `False`):
          The ControlNet encoder tries to recognize the content of the input image even if you remove all
          prompts. A `guidance_scale` value between 3.0 and 5.0 is recommended.
      control_guidance_start (`float` or `List[float]`, *optional*, defaults to 0.0):
          The percentage of total steps at which the ControlNet starts applying.
      control_guidance_end (`float` or `List[float]`, *optional*, defaults to 1.0):
          The percentage of total steps at which the ControlNet stops applying.
      clip_skip (`int`, *optional*):
          Number of layers to be skipped from CLIP while computing the prompt embeddings. A value of 1 means that
          the output of the pre-final layer will be used for computing the prompt embeddings.
      allback_on_step_end (`Callable`, *optional*):
          A function that calls at the end of each denoising steps during the inference. The function is called
          with the following arguments: `callback_on_step_end(self: DiffusionPipeline, step: int, timestep: int,
          callback_kwargs: Dict)`. `callback_kwargs` will include a list of all tensors as specified by
          `callback_on_step_end_tensor_inputs`.
      callback_on_step_end_tensor_inputs (`List`, *optional*):
          The list of tensor inputs for the `callback_on_step_end` function. The tensors specified in the list
          will be passed as `callback_kwargs` argument. You will only be able to include variables listed in the
          `._callback_tensor_inputs` attribute of your pipeine class.
  
  Examples:
  
  Returns:
      [`~pipelines.text_to_video_synthesis.TextToVideoSDPipelineOutput`] or `tuple`:
          If `return_dict` is `True`, [`~pipelines.text_to_video_synthesis.TextToVideoSDPipelineOutput`] is
          returned, otherwise a `tuple` is returned where the first element is a list with the generated frames.
docstr_tok: ["The","call","function","to","the","pipeline","for","generation",".","Args",":","prompt","(","`","str","`","or","`","List","[","str","]","`",",","*","optional","*",")",":","The","prompt","or","prompts","to","guide","image","generation",".","If","not","defined",",","you","need","to","pass","`","prompt_embeds","`",".","height","(","`","int","`",",","*","optional","*",",","defaults","to","`","self.unet.config.sample_size","*","self.vae_scale_factor","`",")",":","The","height","in","pixels","of","the","generated","video",".","width","(","`","int","`",",","*","optional","*",",","defaults","to","`","self.unet.config.sample_size","*","self.vae_scale_factor","`",")",":","The","width","in","pixels","of","the","generated","video",".","num_frames","(","`","int","`",",","*","optional","*",",","defaults","to","16",")",":","The","number","of","video","frames","that","are","generated",".","Defaults","to","16","frames","which","at","8","frames","per","seconds","amounts","to","2","seconds","of","video",".","num_inference_steps","(","`","int","`",",","*","optional","*",",","defaults","to","50",")",":","The","number","of","denoising","steps",".","More","denoising","steps","usually","lead","to","a","higher","quality","videos","at","the","expense","of","slower","inference",".","guidance_scale","(","`","float","`",",","*","optional","*",",","defaults","to","7.5",")",":","A","higher","guidance","scale","value","encourages","the","model","to","generate","images","closely","linked","to","the","text","`","prompt","`","at","the","expense","of","lower","image","quality",".","Guidance","scale","is","enabled","when","`","guidance_scale",">","1","`",".","negative_prompt","(","`","str","`","or","`","List","[","str","]","`",",","*","optional","*",")",":","The","prompt","or","prompts","to","guide","what","to","not","include","in","image","generation",".","If","not","defined",",","you","need","to","pass","`","negative_prompt_embeds","`","instead",".","Ignored","when","not","using","guidance","(","`","guidance_scale","<","1","`",")",".","eta","(","`","float","`",",","*","optional","*",",","defaults","to","0.0",")",":","Corresponds","to","parameter","eta","(","\u03b7",")","from","the","[","DDIM","]","(","https",":","\/\/arxiv.org\/abs\/2010.02502",")","paper",".","Only","applies","to","the","[","`","~schedulers.DDIMScheduler","`","]",",","and","is","ignored","in","other","schedulers",".","generator","(","`","torch.Generator","`","or","`","List","[","torch.Generator","]","`",",","*","optional","*",")",":","A","[","`","torch.Generator","`","]","(","https",":","\/\/pytorch.org\/docs\/stable\/generated\/torch.Generator.html",")","to","make","generation","deterministic",".","latents","(","`","torch.FloatTensor","`",",","*","optional","*",")",":","Pre-generated","noisy","latents","sampled","from","a","Gaussian","distribution",",","to","be","used","as","inputs","for","video","generation",".","Can","be","used","to","tweak","the","same","generation","with","different","prompts",".","If","not","provided",",","a","latents","tensor","is","generated","by","sampling","using","the","supplied","random","`","generator","`",".","Latents","should","be","of","shape","`","(","batch_size",",","num_channel",",","num_frames",",","height",",","width",")","`",".","prompt_embeds","(","`","torch.FloatTensor","`",",","*","optional","*",")",":","Pre-generated","text","embeddings",".","Can","be","used","to","easily","tweak","text","inputs","(","prompt","weighting",")",".","If","not","provided",",","text","embeddings","are","generated","from","the","`","prompt","`","input","argument",".","negative_prompt_embeds","(","`","torch.FloatTensor","`",",","*","optional","*",")",":","Pre-generated","negative","text","embeddings",".","Can","be","used","to","easily","tweak","text","inputs","(","prompt","weighting",")",".","If","not","provided",",","`","negative_prompt_embeds","`","are","generated","from","the","`","negative_prompt","`","input","argument",".","ip_adapter_image","(","`","PipelineImageInput","`",",","*","optional","*",")",":","Optional","image","input","to","work","with","IP","Adapters",".","conditioning_frames","(","`","List","[","PipelineImageInput","]","`",",","*","optional","*",")",":","The","ControlNet","input","condition","to","provide","guidance","to","the","`","unet","`","for","generation",".","If","multiple","ControlNets","are","specified",",","images","must","be","passed","as","a","list","such","that","each","element","of","the","list","can","be","correctly","batched","for","input","to","a","single","ControlNet",".","output_type","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","pil","''","`",")",":","The","output","format","of","the","generated","video",".","Choose","between","`","torch.FloatTensor","`",",","`","PIL.Image","`","or","`","np.array","`",".","return_dict","(","`","bool","`",",","*","optional","*",",","defaults","to","`","True","`",")",":","Whether","or","not","to","return","a","[","`","~pipelines.text_to_video_synthesis.TextToVideoSDPipelineOutput","`","]","instead","of","a","plain","tuple",".","cross_attention_kwargs","(","`","dict","`",",","*","optional","*",")",":","A","kwargs","dictionary","that","if","specified","is","passed","along","to","the","[","`","AttentionProcessor","`","]","as","defined","in","[","`","self.processor","`","]","(","https",":","\/\/github.com\/huggingface\/diffusers\/blob\/main\/src\/diffusers\/models\/attention_processor.py",")",".","controlnet_conditioning_scale","(","`","float","`","or","`","List","[","float","]","`",",","*","optional","*",",","defaults","to","1.0",")",":","The","outputs","of","the","ControlNet","are","multiplied","by","`","controlnet_conditioning_scale","`","before","they","are","added","to","the","residual","in","the","original","`","unet","`",".","If","multiple","ControlNets","are","specified","in","`","init","`",",","you","can","set","the","corresponding","scale","as","a","list",".","guess_mode","(","`","bool","`",",","*","optional","*",",","defaults","to","`","False","`",")",":","The","ControlNet","encoder","tries","to","recognize","the","content","of","the","input","image","even","if","you","remove","all","prompts",".","A","`","guidance_scale","`","value","between","3.0","and","5.0","is","recommended",".","control_guidance_start","(","`","float","`","or","`","List","[","float","]","`",",","*","optional","*",",","defaults","to","0.0",")",":","The","percentage","of","total","steps","at","which","the","ControlNet","starts","applying",".","control_guidance_end","(","`","float","`","or","`","List","[","float","]","`",",","*","optional","*",",","defaults","to","1.0",")",":","The","percentage","of","total","steps","at","which","the","ControlNet","stops","applying",".","clip_skip","(","`","int","`",",","*","optional","*",")",":","Number","of","layers","to","be","skipped","from","CLIP","while","computing","the","prompt","embeddings",".","A","value","of","1","means","that","the","output","of","the","pre-final","layer","will","be","used","for","computing","the","prompt","embeddings",".","allback_on_step_end","(","`","Callable","`",",","*","optional","*",")",":","A","function","that","calls","at","the","end","of","each","denoising","steps","during","the","inference",".","The","function","is","called","with","the","following","arguments",":","`","callback_on_step_end","(","self",":","DiffusionPipeline",",","step",":","int",",","timestep",":","int",",","callback_kwargs",":","Dict",")","`",".","`","callback_kwargs","`","will","include","a","list","of","all","tensors","as","specified","by","`","callback_on_step_end_tensor_inputs","`",".","callback_on_step_end_tensor_inputs","(","`","List","`",",","*","optional","*",")",":","The","list","of","tensor","inputs","for","the","`","callback_on_step_end","`","function",".","The","tensors","specified","in","the","list","will","be","passed","as","`","callback_kwargs","`","argument",".","You","will","only","be","able","to","include","variables","listed","in","the","`","._callback_tensor_inputs","`","attribute","of","your","pipeine","class",".","Examples",":","Returns",":","[","`","~pipelines.text_to_video_synthesis.TextToVideoSDPipelineOutput","`","]","or","`","tuple","`",":","If","`","return_dict","`","is","`","True","`",",","[","`","~pipelines.text_to_video_synthesis.TextToVideoSDPipelineOutput","`","]","is","returned",",","otherwise","a","`","tuple","`","is","returned","where","the","first","element","is","a","list","with","the","generated","frames","."]
code: def __call__(
      self,
      prompt: Union[str, List[str]] = None,
      num_frames: Optional[int] = 16,
      height: Optional[int] = None,
      width: Optional[int] = None,
      num_inference_steps: int = 50,
      guidance_scale: float = 7.5,
      negative_prompt: Optional[Union[str, List[str]]] = None,
      num_videos_per_prompt: Optional[int] = 1,
      eta: float = 0.0,
      generator: Optional[
          Union[torch.Generator, List[torch.Generator]]
      ] = None,
      latents: Optional[torch.FloatTensor] = None,
      prompt_embeds: Optional[torch.FloatTensor] = None,
      negative_prompt_embeds: Optional[torch.FloatTensor] = None,
      ip_adapter_image: Optional[PipelineImageInput] = None,
      conditioning_frames: Optional[List[PipelineImageInput]] = None,
      output_type: Optional[str] = "pil",
      return_dict: bool = True,
      cross_attention_kwargs: Optional[Dict[str, Any]] = None,
      controlnet_conditioning_scale: Union[float, List[float]] = 1.0,
      guess_mode: bool = False,
      control_guidance_start: Union[float, List[float]] = 0.0,
      control_guidance_end: Union[float, List[float]] = 1.0,
      clip_skip: Optional[int] = None,
      callback_on_step_end: Optional[
          Callable[[int, int, Dict], None]
      ] = None,
      callback_on_step_end_tensor_inputs: List[str] = ["latents"],
      **kwargs,
  ):
      r"""
      The call function to the pipeline for generation.
  
      Args:
          prompt (`str` or `List[str]`, *optional*):
              The prompt or prompts to guide image generation. If not defined, you need to pass `prompt_embeds`.
          height (`int`, *optional*, defaults to `self.unet.config.sample_size * self.vae_scale_factor`):
              The height in pixels of the generated video.
          width (`int`, *optional*, defaults to `self.unet.config.sample_size * self.vae_scale_factor`):
              The width in pixels of the generated video.
          num_frames (`int`, *optional*, defaults to 16):
              The number of video frames that are generated. Defaults to 16 frames which at 8 frames per seconds
              amounts to 2 seconds of video.
          num_inference_steps (`int`, *optional*, defaults to 50):
              The number of denoising steps. More denoising steps usually lead to a higher quality videos at the
              expense of slower inference.
          guidance_scale (`float`, *optional*, defaults to 7.5):
              A higher guidance scale value encourages the model to generate images closely linked to the text
              `prompt` at the expense of lower image quality. Guidance scale is enabled when `guidance_scale > 1`.
          negative_prompt (`str` or `List[str]`, *optional*):
              The prompt or prompts to guide what to not include in image generation. If not defined, you need to
              pass `negative_prompt_embeds` instead. Ignored when not using guidance (`guidance_scale < 1`).
          eta (`float`, *optional*, defaults to 0.0):
              Corresponds to parameter eta (η) from the [DDIM](https://arxiv.org/abs/2010.02502) paper. Only applies
              to the [`~schedulers.DDIMScheduler`], and is ignored in other schedulers.
          generator (`torch.Generator` or `List[torch.Generator]`, *optional*):
              A [`torch.Generator`](https://pytorch.org/docs/stable/generated/torch.Generator.html) to make
              generation deterministic.
          latents (`torch.FloatTensor`, *optional*):
              Pre-generated noisy latents sampled from a Gaussian distribution, to be used as inputs for video
              generation. Can be used to tweak the same generation with different prompts. If not provided, a latents
              tensor is generated by sampling using the supplied random `generator`. Latents should be of shape
              `(batch_size, num_channel, num_frames, height, width)`.
          prompt_embeds (`torch.FloatTensor`, *optional*):
              Pre-generated text embeddings. Can be used to easily tweak text inputs (prompt weighting). If not
              provided, text embeddings are generated from the `prompt` input argument.
          negative_prompt_embeds (`torch.FloatTensor`, *optional*):
              Pre-generated negative text embeddings. Can be used to easily tweak text inputs (prompt weighting). If
              not provided, `negative_prompt_embeds` are generated from the `negative_prompt` input argument.
          ip_adapter_image (`PipelineImageInput`, *optional*):
              Optional image input to work with IP Adapters.
          conditioning_frames (`List[PipelineImageInput]`, *optional*):
              The ControlNet input condition to provide guidance to the `unet` for generation. If multiple ControlNets
              are specified, images must be passed as a list such that each element of the list can be correctly
              batched for input to a single ControlNet.
          output_type (`str`, *optional*, defaults to `"pil"`):
              The output format of the generated video. Choose between `torch.FloatTensor`, `PIL.Image` or
              `np.array`.
          return_dict (`bool`, *optional*, defaults to `True`):
              Whether or not to return a [`~pipelines.text_to_video_synthesis.TextToVideoSDPipelineOutput`] instead
              of a plain tuple.
          cross_attention_kwargs (`dict`, *optional*):
              A kwargs dictionary that if specified is passed along to the [`AttentionProcessor`] as defined in
              [`self.processor`](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/attention_processor.py).
          controlnet_conditioning_scale (`float` or `List[float]`, *optional*, defaults to 1.0):
              The outputs of the ControlNet are multiplied by `controlnet_conditioning_scale` before they are added
              to the residual in the original `unet`. If multiple ControlNets are specified in `init`, you can set
              the corresponding scale as a list.
          guess_mode (`bool`, *optional*, defaults to `False`):
              The ControlNet encoder tries to recognize the content of the input image even if you remove all
              prompts. A `guidance_scale` value between 3.0 and 5.0 is recommended.
          control_guidance_start (`float` or `List[float]`, *optional*, defaults to 0.0):
              The percentage of total steps at which the ControlNet starts applying.
          control_guidance_end (`float` or `List[float]`, *optional*, defaults to 1.0):
              The percentage of total steps at which the ControlNet stops applying.
          clip_skip (`int`, *optional*):
              Number of layers to be skipped from CLIP while computing the prompt embeddings. A value of 1 means that
              the output of the pre-final layer will be used for computing the prompt embeddings.
          allback_on_step_end (`Callable`, *optional*):
              A function that calls at the end of each denoising steps during the inference. The function is called
              with the following arguments: `callback_on_step_end(self: DiffusionPipeline, step: int, timestep: int,
              callback_kwargs: Dict)`. `callback_kwargs` will include a list of all tensors as specified by
              `callback_on_step_end_tensor_inputs`.
          callback_on_step_end_tensor_inputs (`List`, *optional*):
              The list of tensor inputs for the `callback_on_step_end` function. The tensors specified in the list
              will be passed as `callback_kwargs` argument. You will only be able to include variables listed in the
              `._callback_tensor_inputs` attribute of your pipeine class.
  
      Examples:
  
      Returns:
          [`~pipelines.text_to_video_synthesis.TextToVideoSDPipelineOutput`] or `tuple`:
              If `return_dict` is `True`, [`~pipelines.text_to_video_synthesis.TextToVideoSDPipelineOutput`] is
              returned, otherwise a `tuple` is returned where the first element is a list with the generated frames.
      """
  
      callback = kwargs.pop("callback", None)
      callback_steps = kwargs.pop("callback_steps", None)
  
      if callback is not None:
          deprecate(
              "callback",
              "1.0.0",
              "Passing `callback` as an input argument to `__call__` is deprecated, consider using `callback_on_step_end`",
          )
      if callback_steps is not None:
          deprecate(
              "callback_steps",
              "1.0.0",
              "Passing `callback_steps` as an input argument to `__call__` is deprecated, consider using `callback_on_step_end`",
          )
  
      controlnet = (
          self.controlnet._orig_mod
          if is_compiled_module(self.controlnet)
          else self.controlnet
      )
  
      # align format for control guidance
      if not isinstance(control_guidance_start, list) and isinstance(
          control_guidance_end, list
      ):
          control_guidance_start = len(control_guidance_end) * [
              control_guidance_start
          ]
      elif not isinstance(control_guidance_end, list) and isinstance(
          control_guidance_start, list
      ):
          control_guidance_end = len(control_guidance_start) * [
              control_guidance_end
          ]
      elif not isinstance(
          control_guidance_start, list
      ) and not isinstance(control_guidance_end, list):
          mult = (
              len(controlnet.nets)
              if isinstance(controlnet, MultiControlNetModel)
              else 1
          )
          control_guidance_start, control_guidance_end = (
              mult * [control_guidance_start],
              mult * [control_guidance_end],
          )
  
      # 0. Default height and width to unet
      height = (
          height or self.unet.config.sample_size * self.vae_scale_factor
      )
      width = (
          width or self.unet.config.sample_size * self.vae_scale_factor
      )
  
      num_videos_per_prompt = 1
  
      # 1. Check inputs. Raise error if not correct
      self.check_inputs(
          prompt=prompt,
          height=height,
          width=width,
          num_frames=num_frames,
          callback_steps=callback_steps,
          negative_prompt=negative_prompt,
          callback_on_step_end_tensor_inputs=callback_on_step_end_tensor_inputs,
          prompt_embeds=prompt_embeds,
          negative_prompt_embeds=negative_prompt_embeds,
          image=conditioning_frames,
          controlnet_conditioning_scale=controlnet_conditioning_scale,
          control_guidance_start=control_guidance_start,
          control_guidance_end=control_guidance_end,
      )
  
      self._guidance_scale = guidance_scale
      self._clip_skip = clip_skip
      self._cross_attention_kwargs = cross_attention_kwargs
  
      # 2. Define call parameters
      if prompt is not None and isinstance(prompt, str):
          batch_size = 1
      elif prompt is not None and isinstance(prompt, list):
          batch_size = len(prompt)
      else:
          batch_size = prompt_embeds.shape[0]
  
      device = self._execution_device
  
      if isinstance(controlnet, MultiControlNetModel) and isinstance(
          controlnet_conditioning_scale, float
      ):
          controlnet_conditioning_scale = [
              controlnet_conditioning_scale
          ] * len(controlnet.nets)
  
      global_pool_conditions = (
          controlnet.config.global_pool_conditions
          if isinstance(controlnet, ControlNetModel)
          else controlnet.nets[0].config.global_pool_conditions
      )
      guess_mode = guess_mode or global_pool_conditions
  
      # 3. Encode input prompt
      text_encoder_lora_scale = (
          cross_attention_kwargs.get("scale", None)
          if cross_attention_kwargs is not None
          else None
      )
      prompt_embeds, negative_prompt_embeds = self.encode_prompt(
          prompt,
          device,
          num_videos_per_prompt,
          self.do_classifier_free_guidance,
          negative_prompt,
          prompt_embeds=prompt_embeds,
          negative_prompt_embeds=negative_prompt_embeds,
          lora_scale=text_encoder_lora_scale,
          clip_skip=self.clip_skip,
      )
      # For classifier free guidance, we need to do two forward passes.
      # Here we concatenate the unconditional and text embeddings into a single batch
      # to avoid doing two forward passes
      if self.do_classifier_free_guidance:
          prompt_embeds = torch.cat(
              [negative_prompt_embeds, prompt_embeds]
          )
  
      if ip_adapter_image is not None:
          image_embeds, negative_image_embeds = self.encode_image(
              ip_adapter_image, device, num_videos_per_prompt
          )
          if self.do_classifier_free_guidance:
              image_embeds = torch.cat(
                  [negative_image_embeds, image_embeds]
              )
  
      if isinstance(controlnet, ControlNetModel):
          conditioning_frames = self.prepare_image(
              image=conditioning_frames,
              width=width,
              height=height,
              batch_size=batch_size
              * num_videos_per_prompt
              * num_frames,
              num_images_per_prompt=num_videos_per_prompt,
              device=device,
              dtype=controlnet.dtype,
              do_classifier_free_guidance=self.do_classifier_free_guidance,
              guess_mode=guess_mode,
          )
      elif isinstance(controlnet, MultiControlNetModel):
          cond_prepared_frames = []
          for frame_ in conditioning_frames:
              prepared_frame = self.prepare_image(
                  image=frame_,
                  width=width,
                  height=height,
                  batch_size=batch_size
                  * num_videos_per_prompt
                  * num_frames,
                  num_images_per_prompt=num_videos_per_prompt,
                  device=device,
                  dtype=controlnet.dtype,
                  do_classifier_free_guidance=self.do_classifier_free_guidance,
                  guess_mode=guess_mode,
              )
              cond_prepared_frames.append(prepared_frame)
          conditioning_frames = cond_prepared_frames
      else:
          assert False
  
      # 4. Prepare timesteps
      self.scheduler.set_timesteps(num_inference_steps, device=device)
      timesteps = self.scheduler.timesteps
      self._num_timesteps = len(timesteps)
  
      # 5. Prepare latent variables
      num_channels_latents = self.unet.config.in_channels
      latents = self.prepare_latents(
          batch_size * num_videos_per_prompt,
          num_channels_latents,
          num_frames,
          height,
          width,
          prompt_embeds.dtype,
          device,
          generator,
          latents,
      )
  
      # 6. Prepare extra step kwargs. TODO: Logic should ideally just be moved out of the pipeline
      extra_step_kwargs = self.prepare_extra_step_kwargs(generator, eta)
  
      # 7. Add image embeds for IP-Adapter
      added_cond_kwargs = (
          {"image_embeds": image_embeds}
          if ip_adapter_image is not None
          else None
      )
  
      # 7.1 Create tensor stating which controlnets to keep
      controlnet_keep = []
      for i in range(len(timesteps)):
          keeps = [
              1.0
              - float(
                  i / len(timesteps) < s or (i + 1) / len(timesteps) > e
              )
              for s, e in zip(
                  control_guidance_start, control_guidance_end
              )
          ]
          controlnet_keep.append(
              keeps[0]
              if isinstance(controlnet, ControlNetModel)
              else keeps
          )
  
      # Denoising loop
      num_warmup_steps = (
          len(timesteps) - num_inference_steps * self.scheduler.order
      )
      with self.progress_bar(total=num_inference_steps) as progress_bar:
          for i, t in enumerate(timesteps):
              # expand the latents if we are doing classifier free guidance
              latent_model_input = (
                  torch.cat([latents] * 2)
                  if self.do_classifier_free_guidance
                  else latents
              )
              latent_model_input = self.scheduler.scale_model_input(
                  latent_model_input, t
              )
  
              if guess_mode and self.do_classifier_free_guidance:
                  # Infer ControlNet only for the conditional batch.
                  control_model_input = latents
                  control_model_input = (
                      self.scheduler.scale_model_input(
                          control_model_input, t
                      )
                  )
                  controlnet_prompt_embeds = prompt_embeds.chunk(2)[1]
              else:
                  control_model_input = latent_model_input
                  controlnet_prompt_embeds = prompt_embeds
              controlnet_prompt_embeds = (
                  controlnet_prompt_embeds.repeat_interleave(
                      num_frames, dim=0
                  )
              )
  
              if isinstance(controlnet_keep[i], list):
                  cond_scale = [
                      c * s
                      for c, s in zip(
                          controlnet_conditioning_scale,
                          controlnet_keep[i],
                      )
                  ]
              else:
                  controlnet_cond_scale = controlnet_conditioning_scale
                  if isinstance(controlnet_cond_scale, list):
                      controlnet_cond_scale = controlnet_cond_scale[0]
                  cond_scale = (
                      controlnet_cond_scale * controlnet_keep[i]
                  )
  
              control_model_input = torch.transpose(
                  control_model_input, 1, 2
              )
              control_model_input = control_model_input.reshape(
                  (
                      -1,
                      control_model_input.shape[2],
                      control_model_input.shape[3],
                      control_model_input.shape[4],
                  )
              )
  
              (
                  down_block_res_samples,
                  mid_block_res_sample,
              ) = self.controlnet(
                  control_model_input,
                  t,
                  encoder_hidden_states=controlnet_prompt_embeds,
                  controlnet_cond=conditioning_frames,
                  conditioning_scale=cond_scale,
                  guess_mode=guess_mode,
                  return_dict=False,
              )
  
              # predict the noise residual
              noise_pred = self.unet(
                  latent_model_input,
                  t,
                  encoder_hidden_states=prompt_embeds,
                  cross_attention_kwargs=self.cross_attention_kwargs,
                  added_cond_kwargs=added_cond_kwargs,
                  down_block_additional_residuals=down_block_res_samples,
                  mid_block_additional_residual=mid_block_res_sample,
              ).sample
  
              # perform guidance
              if self.do_classifier_free_guidance:
                  noise_pred_uncond, noise_pred_text = noise_pred.chunk(
                      2
                  )
                  noise_pred = noise_pred_uncond + guidance_scale * (
                      noise_pred_text - noise_pred_uncond
                  )
  
              # compute the previous noisy sample x_t -> x_t-1
              latents = self.scheduler.step(
                  noise_pred, t, latents, **extra_step_kwargs
              ).prev_sample
  
              if callback_on_step_end is not None:
                  callback_kwargs = {}
                  for k in callback_on_step_end_tensor_inputs:
                      callback_kwargs[k] = locals()[k]
                  callback_outputs = callback_on_step_end(
                      self, i, t, callback_kwargs
                  )
  
                  latents = callback_outputs.pop("latents", latents)
                  prompt_embeds = callback_outputs.pop(
                      "prompt_embeds", prompt_embeds
                  )
                  negative_prompt_embeds = callback_outputs.pop(
                      "negative_prompt_embeds", negative_prompt_embeds
                  )
  
              # call the callback, if provided
              if i == len(timesteps) - 1 or (
                  (i + 1) > num_warmup_steps
                  and (i + 1) % self.scheduler.order == 0
              ):
                  progress_bar.update()
                  if callback is not None and i % callback_steps == 0:
                      callback(i, t, latents)
  
      if output_type == "latent":
          return AnimateDiffControlNetPipelineOutput(frames=latents)
  
      # Post-processing
      video_tensor = self.decode_latents(latents)
  
      if output_type == "pt":
          video = video_tensor
      else:
          video = tensor2vid(
              video_tensor,
              self.image_processor,
              output_type=output_type,
          )
  
      # Offload all models
      self.maybe_free_model_hooks()
  
      if not return_dict:
          return (video,)
  
      return AnimateDiffControlNetPipelineOutput(frames=video)
  
imports: import inspect
  from dataclasses import dataclass
  from typing import Any, Callable, Dict, List, Optional, Tuple, Union
  import numpy
  import torch
  import torch.nn.functional
  from PIL import Image
  from transformers import CLIPImageProcessor, CLIPTextModel, CLIPTokenizer, CLIPVisionModelWithProjection
  from diffusers.image_processor import PipelineImageInput, VaeImageProcessor
  from diffusers.loaders import IPAdapterMixin, LoraLoaderMixin, TextualInversionLoaderMixin
  from diffusers.models import AutoencoderKL, ControlNetModel, UNet2DConditionModel, UNetMotionModel
  from diffusers.models.lora import adjust_lora_scale_text_encoder
  from diffusers.models.unet_motion_model import MotionAdapter
  from diffusers.pipelines.controlnet.multicontrolnet import MultiControlNetModel
  from diffusers.pipelines.pipeline_utils import DiffusionPipeline
  from diffusers.schedulers import DDIMScheduler, DPMSolverMultistepScheduler, EulerAncestralDiscreteScheduler, EulerDiscreteScheduler, LMSDiscreteScheduler, PNDMScheduler
  from diffusers.utils import USE_PEFT_BACKEND, BaseOutput, deprecate, logging, scale_lora_layers, unscale_lora_layers
  from diffusers.utils.torch_utils import is_compiled_module, randn_tensor
label_desc: Use image node_id 21 for calling the AnimateDiffControlNetPipeline obj's underlying member method code with example usage: obj.__call__(prompt, num_frames, height, width, num_inference_steps, guidance_scale, negative_prompt, num_videos_per_prompt, eta, generator, latents, prompt_embeds, negative_prompt_embeds, ip_adapter_image, conditioning_frames, output_type, return_dict, cross_attention_kwargs, controlnet_conditioning_scale, guess_mode, control_guidance_start, control_guidance_end, clip_skip, callback_on_step_end, callback_on_step_end_tensor_inputs) and returns: AnimateDiffControlNetPipelineOutput, AnimateDiffControlNetPipelineOutput, video