TI2V-Zero: Zero-Shot Image Conditioning for Text-to-Video Diffusion Models
Haomiao Ni Bernhard Egger Suhas Lohit Anoop Cherian Ye Wang Toshiaki Koike-Akino Sharon X. Huang Tim K. Marks

Measurement of Interstellar Magnetization by Synchrotron Polarization Variance
    Xue-Juan Yang
    Received March XX, 2024; accepted XX XX, 2024
==============================================================================


Text-conditioned image-to-video generation (TI2V) aims to synthesize a realistic video starting from a given image (, a woman's photo) and a text description (, “a woman is drinking water."). Existing TI2V frameworks often require costly training on video-text datasets and specific model designs for text and image conditioning. In this paper, we propose TI2V-Zero, a zero-shot, tuning-free method that empowers a pretrained text-to-video (T2V) diffusion model to be conditioned on a provided image, enabling TI2V generation without any optimization, fine-tuning, or introducing external modules. Our approach leverages a pretrained T2V diffusion foundation model as the generative prior. To guide video generation with the additional image input, we propose a “repeat-and-slide” strategy that modulates the reverse denoising process, allowing the frozen diffusion model to synthesize a video frame-by-frame starting from the provided image. To ensure temporal continuity, we employ a DDPM inversion strategy to initialize Gaussian noise for each newly synthesized frame and a resampling technique to help preserve visual details. We conduct comprehensive experiments on both domain-specific and open-domain datasets, where TI2V-Zero consistently outperforms a recent open-domain TI2V model. Furthermore, we show that TI2V-Zero can seamlessly extend to other tasks such as video infilling and prediction when provided with more images. Its autoregressive design also supports long video generation.






§ INTRODUCTION

Image-to-video (I2V) generation is an appealing topic with various applications, including artistic creation, entertainment, and data augmentation for machine learning <cit.>. 
Given a single image x^0 and a text prompt y, text-conditioned image-to-video (TI2V) generation aims to synthesize M new frames to yield a realistic video, x̂=⟨ x^0, x̂^1, …, x̂^M ⟩, starting from the given frame x^0 and satisfying the text description y. 
Current TI2V generation methods <cit.> typically rely on computationally-heavy training on video-text datasets and specific architecture designs to enable text and image conditioning.
Some <cit.> are constrained to specific domains due to the lack of training with large-scale open-domain datasets. Other approaches, such as <cit.>, utilize pretrained foundation models to reduce training costs, but they still need to train additional modules using video data.



In this paper, we propose TI2V-Zero, which achieves zero-shot TI2V generation using only an open-domain pretrained text-to-video (T2V) latent diffusion model <cit.>.
Here “zero-shot” means that when using the diffusion model (DM) that was trained only for text conditioning, our framework enables image conditioning without any optimization, fine-tuning, or introduction of additional modules.
Specifically, we guide the generation process by incorporating the provided image x^0 into the output latent code at each reverse denoising step. 
To ensure that the temporal attention layers of the pretrained DM focus on information from the given image, we propose a “repeat-and-slide” strategy to synthesize the video in a frame-by-frame manner, rather than directly generating the entire video volume. 
Notably, TI2V-Zero is not trained for the specific domain of the provided image, thus allowing the model to generalize to any image during inference. Additionally, its autoregressive generation makes the synthesis of long videos possible. 

While the standard denoising sampling process starting with randomly initialized Gaussian noise can produce matching semantics, it often results in temporally inconsistent videos. 
Therefore, we introduce an inversion strategy based on the DDPM <cit.> forward process, to provide a more suitable initial noise for generating each new frame. 
We also apply a resampling technique <cit.> in the video DM to help preserve the generated visual details. Our approach ensures that the network maintains temporal consistency, generating visually convincing videos conditioned on the given starting image (see <ref>). 

We conduct extensive experiments on MUG <cit.>, UCF-101 <cit.>, and a new open-domain dataset. In these experiments, TI2V-Zero consistently performs well, outperforming a state-of-the-art model <cit.> that was based on a video diffusion foundation model <cit.> and was specifically trained to enable open-domain TI2V generation.




§ RELATED WORK



 §.§ Conditional Image-to-Video Generation

Conditional video generation aims to synthesize videos guided by user-provided signals.
It can be classified according to which type(s) of conditions are given, such as text-to-video (T2V) generation <cit.>, video-to-video (V2V) generation <cit.>, and image-to-video (I2V) generation <cit.>. 
Here we discuss previous text-conditioned image-to-video (TI2V) generation methods <cit.>. Hu <cit.> introduced MAGE, a TI2V generator that integrates a motion anchor structure to store appearance-motion-aligned representations through three-dimensional axial transformers. 
Yin <cit.> proposed DragNUWA, a diffusion-based model capable of generating videos controlled by text, image, and trajectory information with three modules including a trajectory sampler, a multi-scale fusion, and an adaptive training strategy.
However, these TI2V frameworks require computationally expensive training on video-text datasets and a particular model design to support text-and-image-conditioned training. 
In contrast, our proposed TI2V-Zero leverages a pretrained T2V diffusion model to achieve zero-shot TI2V generation without additional optimization or fine-tuning, making it suitable for a wide range of applications.  




 §.§ Adaptation of Diffusion Foundation Models

Due to the recent successful application of diffusion models (DM) <cit.> to both image and video generation, visual diffusion foundation models have gained prominence. 
These include text-to-image (T2I) models such as Imagen <cit.> and Stable Diffusion <cit.>, as well as text-to-video (T2V) models such as ModelScopeT2V <cit.> and VideoCrafter1 <cit.>.
These models are trained with large-scale open-domain datasets, often including LAION-400M <cit.> and WebVid-10M <cit.>. 
They have shown immense potential for adapting their acquired knowledge base to address a wide range of downstream tasks, thereby reducing or eliminating the need for extensive labeled data.
For example, previous works have explored the application of large T2I models to personalized image generation <cit.>, image editing <cit.>, image segmentation <cit.>, video editing <cit.>, and video generation <cit.>. 
In contrast to T2I models, there are fewer works on the adaptation of large-scale T2V models. 
Xing <cit.> proposed DynamicCrafter for open-domain TI2V generation by adapting a T2V foundation model <cit.>. To control the generative process, they first employed a learnable image encoding network to project the given image into a text-aligned image embedding space. Subsequently, they utilized dual cross-attention layers to fuse text and image information and also concatenated the image with the initial noise to provide the video DM with more precise image details. 
In contrast, in this paper we explore how to inject the provided image to guide the DM sampling process based solely on the pretrained T2V model itself, with no additional training for the new TI2V task. 






§ METHODOLOGY

Given one starting image x^0 and text y, let 𝐱=⟨ x^0, x^1, …, x^M⟩ represent a real video corresponding to text y.
The objective of text-conditioned image-to-video (TI2V) generation is to synthesize a video 𝐱̂=⟨x^0, x̂^1, …, x̂^M⟩, such that the conditional distribution of 𝐱̂ given x^0 and y is identical to the conditional distribution of 𝐱 given x^0 and y, i.e., p(𝐱̂|x^0, y)=p(𝐱|x^0, y). 
Our proposed TI2V-Zero can be built on a pretrained T2V diffusion model with a 3D-UNet-based denoising network. Here we choose ModelScopeT2V <cit.> as backbone due to its promising open-domain T2V generation ability.
Below, we first introduce preliminaries about diffusion models, then introduce the architecture of the pretrained T2V model, and finally present the details of our TI2V-Zero.



 §.§ Preliminaries: Diffusion Models

Diffusion Models (DM) <cit.> are probabilistic models designed to learn a data distribution. 
Here we introduce the fundamental concepts of Denoising Diffusion Probabilistic Models (DDPM).
Given a sample from the data distribution 𝐳_0∼ q(𝐳_0), the forward diffusion process of a DM produces a Markov chain 𝐳_1, …, 𝐳_T by iteratively adding Gaussian noise to 𝐳_0 according to a variance schedule β_1, …, β_T, that is:


    q(𝐳_t|𝐳_t-1) = 𝒩(𝐳_t; √(1-β_t)𝐳_t-1, β_t𝐈)
    ,


where variances β_t are constant.
When the β_t are small, the posterior q(𝐳_t-1|𝐳_t) can be well approximated by a diagonal Gaussian <cit.>. 
Furthermore, if the length of the chain, denoted by T, is sufficiently large, 𝐳_T can be well approximated by a standard Gaussian distribution 𝒩(0, 𝐈). 
These suggest that the true posterior q(𝐳_t-1|𝐳_t) can be estimated by p_θ(𝐳_t-1|𝐳_t) defined as:


    p_θ(𝐳_t-1|𝐳_t)=𝒩(𝐳_t-1; μ_θ(𝐳_t), σ_t^2𝐈)
    ,


where variances σ_t are also constants.
The reverse denoising process in the DM (also termed sampling) then generates samples 𝐳_0∼ p_θ(𝐳_0) by starting with Gaussian noise 𝐳_T∼𝒩(0, 𝐈) and gradually reducing noise in a Markov chain 𝐳_T-1, 𝐳_T-2, …, 𝐳_0 using a learned p_θ(𝐳_t-1|𝐳_t). 
To learn p_θ(𝐳_t-1|𝐳_t), Gaussian noise ϵ is first added to 𝐳_0 to generate samples 𝐳_t. 
Utilizing the independence property of the noise added at each forward step in <ref>, we can calculate the total noise variance as α_t=∏^t_i=0(1-β_i) and transform 𝐳_0 to 𝐳_t in a single step:


    q(𝐳_t|𝐳_0)= 𝒩(𝐳_t; √(α_t)𝐳_0, (1-α_t)𝐈)
        .


Then a model ϵ_θ is trained to predict ϵ using the following mean-squared error loss:


    L=𝔼_t∼𝒰(1, T), 𝐳_0∼ q(𝐳_0), ϵ∼𝒩(0, 𝐈)[||ϵ-ϵ_θ(𝐳_t, t)||^2]
    ,


where diffusion step t is uniformly sampled from {1, …, T}. 
Then μ_θ(𝐳_t) in <ref> can be derived from ϵ_θ(𝐳_t, t) to model p_θ(𝐳_t-1|𝐳_t) <cit.>.
The denoising model ϵ_θ is implemented using a time-conditioned U-Net <cit.> with residual blocks <cit.> and self-attention layers <cit.>. Diffusion step t is specified to ϵ_θ by the sinusoidal position embedding <cit.>.
Conditional generation that samples  can be achieved by learning a y-conditioned model ϵ_θ(𝐳_t, t, y) <cit.> with classifier-free guidance <cit.>.
During training, the condition y in ϵ_θ(𝐳_t, t, y) is replaced by a null label ∅ with a fixed probability. When sampling, the output is generated as follows:




    ϵ̂_θ(𝐳_t, t, y) = ϵ_θ(𝐳_t, t, ∅) + g · (ϵ_θ(𝐳_t, t, y)-ϵ_θ(𝐳_t, t, ∅))
    ,


where g is the guidance scale.





 §.§ Architecture of Pretrained T2V Model

TI2V-Zero can be built upon a pretrained T2V diffusion model with a 3D-UNet-based denoising network. Here we choose ModelScopeT2V <cit.> as the pretrained model (denoted ℳ). 
We now describe this T2V model in detail.

Structure Overview.  Given a text prompt y, the T2V model ℳ synthesizes a video 𝐱̂=⟨x̂^0, x̂^1, …, x̂^K⟩ with a pre-defined video of length (K+1) using a latent video diffusion model. 
Similar to Latent Diffusion Models (LDM) <cit.>, ℳ incorporates a frame auto-encoder <cit.> for the conversion of data between pixel space 𝒳 and latent space 𝒵 through its encoder ℰ and decoder 𝒟. 
Given the real video 𝐱=⟨ x^0, x^1, …, x^K⟩, ℳ first utilizes the frame encoder ℰ to encode the video 𝐱 as 𝐳=⟨ z^0, z^1, …, z^K⟩. 
Here the sizes of pixel frame x and latent frame z are H_x× W_x× 3 and H_z× W_z× C_z, respectively. 
To be consistent with the notation used for the DM, we denote the clean video latent 𝐳=𝐳_0=⟨ z_0^0, z_0^1, …, z_0^K⟩.
ℳ then learns a DM on the latent space 𝒵 through a 3D denoising U-Net ϵ_θ <cit.>. 
Let 𝐳_t=⟨z_t^0, z_t^1, …, z_t^K⟩ represent the latent sequence that results from adding noise over t steps to the original latent sequence 𝐳_0.
When training, the forward diffusion process of a DM transforms the initial latent sequence 𝐳_0 into 𝐳_T by iteratively adding Gaussian noise ϵ for T steps. 
During inference, denoising U-Net ϵ_θ predicts the added noise at each step, enabling the generation of the clean latent sequence 𝐳̂_0=⟨ẑ_0^0, ẑ_0^1, …, ẑ_0^K⟩ starting from randomly sampled Gaussian noise 𝐳_T∼𝒩(0, 𝐈).

Text Conditioning Mechanism. 
ℳ employs a cross-attention mechanism <cit.> to incorporate text information into the generative process as guidance.
Specifically, ℳ uses a pretrained CLIP model <cit.> to encode the prompt y as the text embedding e. The embedding e is later used as the key and value in the multi-head attention layer within the spatial attention blocks, thus enabling the integration of text features with the intermediate U-Net features in ϵ_θ.

Denoising U-Net. 
The denoising U-Net ϵ_θ includes four key building blocks: the initial block, the downsampling block, the spatio-temporal block, and the upsampling block.
The initial block transfers the input into the embedding space, while the downsampling and upsampling blocks are responsible for spatially downsampling and upsampling the feature maps. 
The spatio-temporal block is designed to capture spatial and temporal dependencies in the latent space, which comprises 2D spatial convolution, 1D temporal convolution, 2D spatial attention, and 1D temporal attention.





 §.§ Our Framework

Leveraging the pretrained T2V foundation model ℳ, we first propose a straightforward replacing-based baseline for adapting ℳ to TI2V generation. We then analyze the possible reasons why it fails and introduce our TI2V-Zero framework, which includes a repeat-and-slide strategy, DDPM-based inversion, and resampling. Figure <ref> and <ref> demonstrate the inference process of TI2V-Zero.

Replacing-based Baseline.
We assume that the pretrained model ℳ is designed to generate the video with a fixed length of (K+1). So we first consider synthesizing videos with that same length (K+1), i.e., M=K.
Since the DM process operates within the latent space 𝒵, we use the encoder ℰ to map the given starting frame x^0 into the latent representation z^0. 
Additionally, we denote z^0=z^0_0 to specify that the latent is clean and corresponds to diffusion step 0 of the DM.
Note that each reverse denoising step in <ref> from 𝐳̂_t to 𝐳̂_t-1 depends solely on 𝐳̂_t=⟨ẑ_t^0, ẑ_t^1, …, ẑ_t^K⟩.
To ensure that the first frame of the final synthesized clean video latent 𝐳̂_0=⟨ẑ_0^0, ẑ_0^1, …, ẑ_0^K⟩ at step 0 matches the provided image latent, i.e., ẑ_0^0=z^0_0,
we can modify the first generated latent ẑ^0_t of 𝐳̂_t at each reverse step, as long as the signal-to-noise ratio of each frame latent in 𝐳̂_t remains consistent.
Using <ref>, we can add t steps of noise to the provided image latent z^0_0, allowing us to sample z^0_t through a single-step calculation.
By replacing the first generated latent ẑ^0_t with the noisy image latent z^0_t at each reverse denoising step, we might expect that the video generation process can be guided by z^0_0 with the following expressions defined for each reverse step:



    
    z_t^0∼𝒩(√(α)_tz^0_0, (1-α_t)𝐈)
        , 
       ẑ_t^0z_t^0
        , 
       𝐳̂_t-1∼𝒩(μ_θ(𝐳̂_t, y), σ_t^2𝐈).



Specifically, in each reverse step from 𝐳̂_t to 𝐳̂_t-1, as shown in <ref>, we first compute the noisy latent z_t^0 by adding Gaussian noise to the given image latent z_0^0 over t steps. 
Then, we replace the first latent ẑ_t^0 of 𝐳̂_t with z_t^0 in <ref> to incorporate the provided image into the generation process.
Finally, in <ref>, we pass 𝐳̂_t through the denoising network to generate 𝐳̂_t-1, where the text y is integrated by classifier-free guidance (<ref>).
After T iterations, the final clean latent 𝐳̂_0 at diffusion step 0 can be mapped back into the image space 𝒳 using the decoder 𝒟.

Using this replacing-based baseline, we might expect that the temporal attention layers in ϵ_θ can utilize the context provided by the first frame latent ẑ_t^0 to generate the subsequent frame latents in a manner that harmonizes with ẑ_t^0.
However, as shown in Fig. <ref>, row 2, this replacing-based approach fails to produce a video that is temporally consistent with the first image. The generated frames are consistent with each other, but not with the provided first frame. 

To analyze possible reasons for failure, we apply this baseline to a simpler video infilling task, where every other frame is provided and the model needs to predict the interspersed frames.
In this case, the baseline replaces the generated frame latents at positions corresponding to real frames with noisy provided-frame latents in each reverse step.
The resulting video, in <ref>, row 3, looks like a combination of two independent videos: the generated (even) frames are consistent with each other but not with the provided (odd) frames.
We speculate that this may result from the intrinsic dissimilarity between frame latents derived from the given real images and those sampled from ϵ_θ.
Thus, the temporal attention values between frame latents sampled in the same way (both from the given images or both from ϵ_θ) will be higher, while the attention values between frame latents sampled in different ways (one from the given image and the other from ϵ_θ) will be lower. 
Therefore, the temporal attention layers of ℳ tend to utilize the information from latents produced by ϵ_θ to synthesize new frames at each reverse step, ignoring the provided frames. 
We further simplify the task to single-frame prediction, where the model only needs to predict a single frame when all the other frames in the video are given. 
In this setting, all the frame latents except for the final frame are replaced by noisy provided-frame latents in each reverse step. Thus, temporal attention layers can only use information from the real frames.
In this case, <ref>, row 4, shows that the baseline can now generate a final frame that is consistent with the previous frames.



Repeat-and-Slide Strategy. Inspired by the observation in <ref>, to guarantee that the temporal attention layers of ℳ depend solely on the given image, we make two major changes to the proposed replacing-based baseline: 
(1) instead of using ℳ to directly synthesize the entire (K+1)-frame video, we switch to a frame-by-frame generation approach, i.e., we generate only one new frame latent in each complete DM sampling process; 
(2) for each sampling process generating the new frame latent, we ensure that only one frame latent is produced from ϵ_θ, while the other K frame latents are derived from the given real image and previously synthesized frames, thereby forcing temporal attention layers to only use the information from these frame latents. 
Specifically, we construct a queue of K frame latents, denoted as 𝐬_0=⟨ s^0_0, s^1_0, ⋯, s^K-1_0⟩. We also define  𝐬_t=⟨ s^0_t, s^1_t, ⋯, s^K-1_t⟩, which is obtained by adding t steps of Gaussian noise to the clean 𝐬_0. Similar to our replacing-based baseline in the single-frame prediction task, in each reverse step from 𝐳̂_t to 𝐳̂_t-1, we replace the first K frame latents in 𝐳̂_t by 𝐬_t.
Consequently, the temporal attention layers have to utilize information from 𝐬_0 to synthesize the new frame's latent, ẑ^K_0.
Considering that only one starting image latent z^0 is provided, we propose a “repeat-and-slide” strategy to construct 𝐬_0. At the beginning of video generation, we repeat z^0 for K frames to form 𝐬_0, and gradually perform a sliding operation within the queue 𝐬_0 by dequeuing the first frame latent s^0_0 and enqueuing the newly generated latent ẑ^K_0 after each complete DM sampling process. 
Note that though the initial 𝐬_0 is created by repeating z^0, the noise added to get 𝐬_t is different for each frame's latent in 𝐬_t, thus ensuring diversity.
The following expressions define one reverse step in the DM sampling process:



    
    𝐬_t∼𝒩(√(α)_t𝐬_0, (1-α_t)𝐈), 
       ⟨ẑ_t^0, ẑ_t^1, ⋯, ẑ_t^K-1⟩𝐬_t,
       𝐳̂_t-1∼𝒩(μ_θ(𝐳̂_t, y), σ_t^2𝐈).



Specifically, in each reverse denoising step from 𝐳̂_t to 𝐳̂_t-1, we first add t steps of Gaussian noise to the queue 𝐬_0 to yield 𝐬_t in <ref>. Subsequently, we replace the previous K frames of 𝐳̂_t with 𝐬_t in <ref> and input 𝐳̂_t to the denoising network to produce the less noisy latent 𝐳̂_t-1 (<ref>).

With the repeat-and-slide strategy, model ℳ is tasked with predicting only one new frame, while the preceding K frames are incorporated into the reverse process to ensure that the temporal attention layers depend solely on information derived from the provided image. 

DDPM-based Inversion. Though the DM sampling process starting with randomly sampled Gaussian noise produces matching semantics, the generated video is often temporally inconsistent (<ref>, row 2). 
To provide initial noise that can produce more temporally consistent results, we introduce an inversion strategy based on the DDPM <cit.> forward process when generating the new frame latent.
Specifically, at the beginning of each DM sampling process to synthesize the new frame latent ẑ^K_0, instead of starting with the 𝐳̂_T randomly sampled from 𝒩(0, 𝐈), we add T full steps of Gaussian noise to 𝐬_0 to obtain 𝐬_T using <ref>. 
Note that 𝐳̂ has K+1 frames, while 𝐬 has K frames.
We then use 𝐬_T to initialize the first K frames of 𝐳̂_T. 
We copy the last frame s^K-1_T of 𝐬_T to initialize the final frame ẑ^K_T, as the (K-1)th frame is the closest to the Kth frame.

Resampling. Similar to <cit.>, we further apply a resampling technique, which was initially designed for the image inpainting task, to the video DM to enhance motion coherence. 
Particularly, after performing a one-step denoising operation in the reversed process, we add one-step noise again to revert the latent. This procedure is repeated multiple times for each diffusion step, ensuring harmonization between the predicted and conditioning frame latents (see <ref> for details).











§ EXPERIMENTS



 §.§ Datasets and Metrics

We conduct comprehensive experiments on three datasets. More details about datasets, such as selected subjects and text prompts, can be found in our Supplementary Materials.

MUG facial expression dataset <cit.> contains 1,009 videos of 52 subjects performing 7 different expressions. We include this dataset to evaluate the performance of models in scenarios with small motion and a simple, unchanged background. 
To simplify the experiments, we randomly select 5 male and 5 female subjects, and 4 expressions.
We use the text prompt templates like “” to change the expression class label to be text input. Since the expressions shown in the videos of MUG are often not obvious, we add “slight” in the text input to avoid large motion. 

UCF101 action recognition dataset <cit.> contains 13,320 videos from 101 human action classes. We include this dataset to measure performance under complicated motion and complex, changing backgrounds. To simplify the experiments, we select 10 action classes and the first 10 subjects within each class.
We use text prompt templates such as “.” to change the class label to text input. 

In addition to the above two datasets, we create an OPEN dataset to assess the model's performance in open-domain TI2V generation. We first utilize ChatGPT <cit.> to generate 10 text prompts. Subsequently, we employ Stable Diffusion 1.5 <cit.> to synthesize 100 images from each text prompt, generating a total of 1,000 starting images and 10 text prompts for evaluating TI2V models. 

Data Preprocessing. We resize all the videos/images to 256× 256 resolution. For UCF101, since most of the video frames are not square, we crop the central part of the frames. To obtain ground truth videos for computing metrics, we uniformly sample 16 frames from each video in the datasets to generate the video clips with a fixed length.  

Metrics. 
Following prior work <cit.>, 
we assess the visual quality, temporal coherence, and sample diversity of generated videos using Fréchet Video Distance (FVD) <cit.>.
Similar to Fréchet Inception Distance (FID) <cit.>, which is used for image quality evaluation, FVD utilizes a video classification network I3D <cit.> pretrained on Kinetics-400 dataset <cit.> to extract feature representation of real and synthesized videos. 
Then it calculates the Fréchet distance between the distributions of the real and synthesized video features. 
To measure how well a generated video aligns with the text prompt y (condition accuracy) and the given image x_0 (subject relevance), following <cit.>, we design two variants of FVD, namely text-conditioned FVD (tFVD) and subject-conditioned FVD (sFVD). 
tFVD and sFVD compare the distance between real and synthesized video feature distributions under the same text y or the same subject image x_0, respectively. 
We first compute tFVD and sFVD for each condition y and image x_0, then report their mean and variance as final results.
In our experiments, we generate 1,000 videos for all the models to estimate the feature distributions. 
We compute both tFVD and sFVD on the MUG dataset, but for UCF101, we only consider tFVD since it doesn't contain videos of different actions for the same subject. 
For the OPEN dataset, we only present qualitative results due to the lack of ground truth videos. Unless otherwise specified, all the generated videos are 16 frames (i.e., M=15) with resolution 256× 256.



 §.§ Implementation Details


Model Implementation. 
We take the ModelScopeT2V 1.4.2 <cit.> as basis and implement our modifications. 
For text-conditioned generation, we employ classifier-free guidance with g=9.0 in <ref>. Determined by our preliminary experiments, we choose 10-step DDIM and 4-step resampling as the default setting for MUG and OPEN, and 50-step DDIM and 2-step resampling for UCF101.

Implementation of SOTA Model. We compare our TI2V-Zero with a state-of-the-art (SOTA) model DynamiCrafter, a recent open-domain TI2V framework <cit.>. DynamiCrafter is based on a large-scale pretrained T2V foundation model VideoCrafter1 <cit.>. It introduces a learnable projection network to enable image-conditioned generation and then fine-tunes the entire framework. We implement DynamiCrafter using their provided code with their default settings. 
For a fair comparison, all the generated videos are centrally-cropped and resized to 256× 256. 



 §.§ Result Analysis

Ablation Study. We conduct ablation study of different sampling strategies on MUG. As shown in <ref> and <ref>, compared with generating using randomly sampled Gaussian noise, initializing the input noise with DDPM inversion is important for generating temporally continuous videos, improving all of the metrics dramatically.
For MUG, increasing the DDIM sampling steps from 10 to 50 does not enhance the video quality but requires more inference time. 
Thus, we choose 10-step DDIM as the default setting on MUG. 
As shown in <ref> and <ref>, adding resampling helps preserve identity details (, hairstyle and facial appearance), resulting in lower FVD scores. Increasing resampling steps from 2 to 4 further improves FVD scores. 



Effect of Real/Synthesized Starting Frames. We also explore the effect of video generation starting with real or synthesized frames on UCF101. 
We initially use the first frame of the real videos to generate videos with our TI2V-Zero, termed TI2V-Zero-Real.
Additionally, we utilize the backbone model ModelScopeT2V <cit.> to generate synthetic videos using the text inputs of UCF101. We then employ TI2V-Zero to create videos from the first frame of the generated fake videos, denoted as TI2V-Zero-Fake. 
As shown in <ref>, [TI2V-Zero-Fake vs. ModelScopeT2V] can achieve better FVD scores than [TI2V-Zero-Real vs. Real Videos]. 
The reason may be that frames generated by ModelScopeT2V can be considered as in-distribution data since TI2V-Zero is built upon it.
We also compare the output video distribution of TI2V-Zero-Fake and ModelScopeT2V with real videos in <ref>. Though starting from the same synthesized frames, TI2V-Zero-Fake can generate more realistic videos than the backbone model.

Comparison with SOTA Model. We compare our proposed TI2V-Zero with DynamiCrafter <cit.> in <ref> and <ref>. From <ref>, one can find that DynamiCrafter struggles to preserve details from the given image, and the motion of its generated videos is also less diverse. Note that DynamiCrafter requires additional fine-tuning to enable TI2V generation. In contrast, without using any fine-tuning or introducing external modules, our proposed TI2V-Zero can precisely start with the given image and output more visually-pleasing results, thus achieving much better FVD scores on both MUG and UCF101 datasets in <ref>. 
The comparison between our TI2V-Zero models with and without using resampling in <ref> and <ref> also demonstrates the effectiveness of using resampling, which can help maintain identity and background details. 

Extension to Other Applications. TI2V-Zero can also be extended to other tasks as long as we can construct 𝐬_0 with K images at the beginning. These images can be obtained either from ground truth videos or by applying the repeating operation. Then we can slide 𝐬_0 when generating the subsequent frames. We have applied TI2V-Zero in video infilling (see the last row in <ref>), video prediction (see Supplementary Materials), and long video generation (see <ref>). As shown in <ref>, when generating a 128-frame video on the OPEN dataset, our method can preserve the mountain shape in the background, even at the 71st frame (frame x̂^70). The generated video examples and additional experimental results are in our Supplementary Materials.



§ CONCLUSION

In this paper, we propose a zero-shot text-conditioned image-to-video framework, TI2V-Zero, to generate videos by modulating the sampling process of a pretrained video diffusion model without any optimization or fine-tuning. Comprehensive experiments show that TI2V-Zero can achieve promising performance on multiple datasets.

While showing impressive potential, our proposed TI2V-Zero still has some limitations. First, as TI2V-Zero relies on a pretrained T2V diffusion model, the generation quality of TI2V-Zero is constrained by the capabilities and limitations of the pretrained T2V model. We plan to extend our method to more powerful video diffusion foundation models in the future. Second, our method sometimes generates videos that are blurry or contain flickering artifacts. One possible solution is to apply post-processing methods such as blind video deflickering <cit.> or image/video deblurring <cit.> to enhance the quality of final output videos or the newly synthesized frame in each generation. Finally, compared with GAN and standard video diffusion models, our approach is considerably slower because it requires running the entire diffusion process for each frame generation. We will investigate some faster sampling methods <cit.> to reduce generation time.


    ieeenat_fullname
    





