IV-mixed Sampler: Leveraging Image Diffusion Models for Enhanced Video Synthesis
IV-mixed Sampler;Video Synthesis;Inference-heavy Algorithms;Training-free
Yes
No ethics review needed.
* What is the variance of the quantitative metrics (e.g. FVD in Tables 1 and 2)? The values are pretty close and it’s unclear  if the results are statistically significant
* It would be nice to have more video samples to look at (e.g. on an anonymous website). There is only one set of videos in the supplementary, and it is hard to see motion for ones in the paper appendix.
* Does this method still work in scenarios using distilled models? E.g. 1 step or 2 step generation
* The paper is generally clear, and well written
* The proposed method is novel and interesting
* Results seem to show good improvement in generation quality of the videos, while retaining consistent motion
* Main experiments and ablations are thorough and show the benefits and tradeoffs of different instantiations of the proposed method
The paper presents a novel sampling scheme for video diffusion models that leverages pre-train image diffusion models, as they generally have higher visual fidelity compared to current open source video models. The sampling method takes steps forward / backward (DDIM inversion / DDIM) in the diffusion model, alternating between the score functions of the image and video models. Results show high fidelity visual quality while retaining consistent motion in generated video.
* The sampling process requires both I/V models to be in the same underlying latent space, which may be restrictive. How would this method also be used in cases where there is temporal downsampling in the video latent space (as this this is a very common video generation architecture)?
* It is unclear how useful / relevant this method may be ~6+ months from now, as the main motivation of the paper is leveraging the lack of good open source video models, and public video datasets will get better (e.g. OpenVid10M [1]), and better video generation models will be released (e.g. Mochi-1 [2] as of recently).

[1] https://huggingface.co/datasets/nkp37/OpenVid-1M
[2] https://huggingface.co/genmo/mochi-1-preview
Yes
No ethics review needed.
See weakness
+ The paper combines theoretical proof with practical implementations, and is relatively well-rounded
+ The proposed method seems to make intuitive sense
The paper proposes IV-Mixed Sampler, which is to combine image diffusion models (IDMs) with video diffusion models (VDMs) to get the benefit of higher image quality of IDM and good temporal consistency of VDM. This is achieved by sampling from x_t to x_{t+∆t} and from x_{t+∆t} to x_t using IDM and VDM
The writing of the paper can be improved. Here are several comments but not just limited to these points. Overall the presentation is a bit confusing.
- Why do the paper keep mentioning OpenAI's strawberry in Abstract and Intro, which is closed-sourced and no paper or technical detais was released? How is this important or even related to motivating the paper?
- Figure 2 is confusing and should not be put in the intro. This is more like ablation studies, and it is made rather confusing, especially authors also start to talk about "R-" in the intro out of nowhere (line 94), and the starts to discussion "I-" and "V-" in line 107 without any explanation
- The use of "go", "back", "begin", "end" etc in the equations are also confusing
- Plots like Fig 6 are also hard to read

The results of the model is decent but not very strong. For example, FreeInit is significantly better than ours for UMT-FVD on ModelScope-T2V
Yes
No ethics review needed.
1. What are the detailed reasons that we can't use different VAEs for IDM and VDM when adapting the proposed method? Do you have experiment results about this?
2. How does the IV-Mixed Sampler perform as the length and complexity of the video sequences increase? Are there any scalability issues that the authors have identified?
3. Potential Limitations: Are there any specific scenarios or types of video content where the IV-Mixed Sampler might underperform? If so, how might these limitations be addressed?
Originality: The concept of combining IDMs and VDMs to enhance video synthesis is innovative. The paper addresses a significant gap in the field by improving the visual quality of synthesized videos while preserving temporal coherence.

Quality: The paper is well-structured, with a clear problem statement, methodology, experimental validation, and conclusion. The theoretical analysis provides a solid foundation for the proposed algorithm.

Clarity: The authors have done an excellent job of explaining complex concepts in a clear and concise manner. The figures and tables are well-designed and aid in understanding the content.
The paper introduces IV-Mixed Sampler, a novel algorithm designed to enhance video synthesis by leveraging the strengths of both Image Diffusion Models (IDMs) and Video Diffusion Models (VDMs). The core innovation is the use of IDMs to improve the quality of individual video frames while maintaining temporal coherence through VDMs during the sampling process. The authors claim state-of-the-art performance on four benchmarks, including UCF-101-FVD, MSR-VTT-FVD, Chronomagic-Bench-150, and Chronomagic-Bench-1649. The paper also provides a theoretical analysis of the IV-Mixed Sampler and its transformation into a standard inverse ODE process, as well as an exploration of the design space for hyperparameters.
Evaluation Metrics: The paper employs benchmarks like UCF-FVD and MSR-VTT-FVD, which can not accurately assess text-to-video generation models. More suitable benchmarks like EvalCrafter or Vbench are needed for a comprehensive evaluation.

Demonstration Insufficiency: The provided demos do not clearly demonstrate the proposed method's superiority over baselines, requiring more compelling examples to showcase improvements.

Illustration Clarity: Figure 3's illustrations are not clear. The images presented do not effectively convey the differences in quality and temporal coherence, making it difficult for readers to grasp the paper's points regarding the performance of various samplers. High-quality, clear visualizations are crucial for helping readers understand the nuances of video synthesis methods, and the paper would benefit from improved clarity in its visual aids.

Limited Applicability: The method's compatibility with state-of-the-art video generation models, which often use distinct VAEs from image models, is limited. This restricts its potential applications and prospects in the field.
We design a novel training-free algorithm IV-mixed Sampler that leverages the strengths of image diffusion models to assist video diffusion models surpass their current capabilities.
@inproceedings{
anonymous2024ivmixed,
title={{IV}-mixed Sampler: Leveraging Image Diffusion Models for Enhanced Video Synthesis},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=ImpeMDJfVL},
note={under review}
}
The multi-step sampling mechanism, a key feature of visual diffusion models, has significant potential to replicate the success of OpenAI's Strawberry in enhancing performance by increasing the inference computational cost. Sufficient prior studies have demonstrated that correctly scaling up computation in the sampling process can successfully lead to improved generation quality, enhanced image editing, and compositional generalization. While there have been rapid advancements in developing inference-heavy algorithms for improved image generation, relatively little work has explored inference scaling laws in video diffusion models (VDMs). Furthermore, existing research shows only minimal performance gains that are perceptible to the naked eye. To address this, we design a novel training-free algorithm _IV-Mixed Sampler_ that leverages the strengths of image diffusion models (IDMs) to assist VDMs surpass their current capabilities. The core of _IV-Mixed Sampler_ is to use IDMs to significantly enhance the quality of each video frame and VDMs ensure the temporal coherence of the video during the sampling process. Our experiments have demonstrated that _IV-Mixed Sampler_ achieves state-of-the-art performance on 4 benchmarks including UCF-101-FVD, MSR-VTT-FVD, Chronomagic-Bench-150, and Chronomagic-Bench-1649. For example, the open-source Animatediff with _IV-Mixed Sampler_ reduces the UMT-FVD score from 275.2 to 228.6, closing to 223.1 from the closed-source Pika-2.0.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
IV-mixed Sampler Video Synthesis Inference-heavy Algorithms Training-free
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/0a2308f4e31357d068a5862e1ed00dafc43064a6.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/d69b3f0d64de2fe58227f4192a269761eaf8e6a1.zip
IV-mixed Sampler: Leveraging Image Diffusion Models for Enhanced Video Synthesis
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission