DiffIR2VR-Zero: Zero-Shot Video Restoration with Diffusion-based Image Restoration Models
Video restoration;Zero-shot;Training-free;Diffusion models
Yes
No ethics review needed.
See weaknesses.
1. The method is training-free which makes it computationally practical.
This paper proposes a method for zero-shot video restoration using pre-trained image restoration diffusion models. It combines a hierarchical latent warping strategy for keyframes and local frames and token merging that integrates spatial information, optical flow, and feature-based matching.  The method acheives competitive results on zero-shot video restoration tasks such as super-resolution and denoising.
1. To my understanding, the method relies on certain architectural heuristics inspired by the video-editing literature rather than on a solid mathematical framework. Since the method involves no training, there is no theoretical guarantee that it won't fail under certain conditions.

2. The method combines existing ideas from video editing, which on my opinion is still acceptable; however, this limits its novelty.

3. The improvement over the baseline on some datasets, particularly in the case of denoising, is not significant.
Yes
No ethics review needed.
- Degradations that are not encountered during optical flow training may introduce significant errors. How does the proposed method address this issue, and what motivated the choice of the GMFlow network for this approach?
- In Equation (5), is there a specific reason for including f_{src -> tar} (X(T_{src})) twice? 
- Including additional real-world datasets (e.g., [1, 2]) in the evaluation would help assess the method's generalization and robustness. 
- Could you provide a comparison of inference times with other methods?

Reference

[1] Towards Real-world Video Face Restoration: A New Benchmark, cvprw 2024

[2] Toward convolutional blind denoising of real photographs, cvpr 2019
- The paper is well-structured, making the methodology and findings easy to understand.
- The method achieves competitive results without requiring additional training.
This paper presents a zero-shot video restoration method using pre-trained image restoration diffusion models. The proposed approach introduces a hierarchical latent warping strategy and a hybrid flow-guided token merging approach to enhance both temporal consistency and visual detail. Experimental results demonstrate competitive performance in video super-resolution, denoising, showing advantages over state-of-the-art methods. An ablation study further highlights the importance of each proposed component.
- Both the latent warping and hybrid flow-guided token merging approaches rely heavily on optical flow information, which could be a limitation in cases where optical flow estimation is challenging or inaccurate.
- The paper's novelty is somewhat limited, as it primarily combines two existing methodologies with minor modifications. Specifically, the contributions include adjusting the range of warping frames at global and local levels and introducing a flow-guided confidence criterion for token merging.
Yes
No ethics review needed.
1. VidToMe is implemented on top of Stable Diffusion. In Table 1, the proposed approach (based on Stable Diffusion (SD x4)) is compared with VidToMe, but it does not seem to demonstrate performance gains over VidToMe in terms of objective image quality (PSNR/SSIM) or temporal consistency (E_warp). On the other hand, perceptual quality metrics, such as LPIPS, show improved results. Could you clarify if there is a specific reason why perceptual quality metrics like LPIPS perform better? If there is a trade-off involved, would it be possible to show how performance varies with adjustments to the hyperparameters?
2. Although the proposed method emphasizes compatibility with any 2D image restoration diffusion model, the comparison results in Table 1—before and after applying the proposed method on SD x4—show only modest performance gains, whereas the improvements over the DiffBIR baseline are more substantial. Could you please clarify if there is a specific reason for this difference or if there is any dependence on specific network architecture?
3. It appears that the code for Upscale-A-Video is now available. Since this approach also aims to improve temporal consistency across video frames generated by image generation models, it would be beneficial to include a comparison of the results. If the code is still unavailable, could you instead provide results on the datasets used in Upscale-A-Video and compare them with the figures reported in the Upscale-A-Video manuscript?
The primary contribution of this approach is its ability to leverage conventional image generation models directly, without requiring modifications to network architecture or the need for retraining or fine-tuning. This is achieved through a straightforward yet effective technique: hierarchical token merging within the latent space, which ensures temporal consistency across generated video frames. By using this token merging strategy, the method successfully adapts static image models for dynamic video generation, maintaining coherence between frames without compromising quality. Furthermore, this approach reflects a well-engineered integration of recent advances in token merging, incorporating techniques from VidToMe and Upscale-A-Video. These advancement enhances the network’s capacity to address various video restoration challenges, effectively overcoming the limitations of traditional methods that struggle to maintain temporal consistency across frames.
This paper introduces a versatile zero-shot video restoration approach leveraging pre-trained image restoration diffusion models. Unlike conventional methods that require retraining and often struggle with generalization, this method uses a hierarchical latent warping strategy for keyframes and local frames, along with a hybrid correspondence mechanism that merges tokens based on optical flow and token similarity. This approach demonstrates outstanding zero-shot performance, effectively generalizing across varied datasets and managing extreme degradations, including video super-resolution, denoising, and depth completion. Compatible with numerous 2D restoration diffusion models, the technique is validated through quantitative and visual benchmarks on challenging datasets, removing the need for extensive retraining.
First, the paper’s structure requires improvement, as the current organization makes it challenging to follow. 
A major concern is the lack of a comprehensive comparison between this approach and conventional methods, such as VidToMe and Upscale-A-Video. VidToMe introduces local and global token merging techniques, while Upscale-A-Video presents a flow-based merging approach—both key contributions relevant to this work. Please clarify how this approach differentiates itself from these methods, and refer to the “Questions” section for further details.
Yes
No ethics review needed.
1. Is there a comparison of inference efficiency with baselines?
1. The proposed method is pioneering in achieving zero-shot video restoration by leveraging pre-trained image restoration diffusion models, enabling multiple video restoration tasks without additional training.
2. The method presents strong quantitative and visual results compared with state-of-the-art methods, balancing temporal consistency and detail generation.
This paper proposes a training-free method to leverage pre-trained image restoration diffusion models for zero-shot video restoration. Specifically, the proposed method employs hierarchical latent warping and an enhanced token merging strategy to maintain temporal consistency and restore details across video frames. Experimental results demonstrate the versatility of the method across various tasks, including denoising, super-resolution, and depth estimation.
1. In line 267, the authors claim that the combination of hierarchical latent warping and hybrid flow-guided spatial-aware token merging could achieve adaptation to various degradation types. However, it is not sufficiently discussed that why the combination could handle various degradation types.
2. While the use of optical flow guidance aligns with previous video restoration works [1,2], the paper introduces similarity-based guidance to capture correspondences distinct from optical flow. However, the specific benefits of similarity-based guidance over optical flow are not thoroughly discussed.
3. Although recent video restoration methods, such as Shift-Net and FMA-Net, are included as baselines, some classic methods like BasicVSR++ and RVRT are not compared in the experiments.

[1] Chan K C K, Zhou S, Xu X, et al. Basicvsr++: Improving video super-resolution with enhanced propagation and alignment[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 5972-5981.

[2] Liang J, Fan Y, Xiang X, et al. Recurrent video restoration transformer with guided deformable attention[J]. Advances in Neural Information Processing Systems, 2022, 35: 378-393.
@inproceedings{
anonymous2024diffirvrzero,
title={Diff{IR}2{VR}-Zero: Zero-Shot Video Restoration with Diffusion-based Image Restoration Models},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=qpDqO7qa3R},
note={under review}
}
This paper introduces a method for zero-shot video restoration using pre-trained image restoration diffusion models. Traditional video restoration methods often need retraining for different settings and struggle with limited generalization across various degradation types and datasets. Our approach uses a hierarchical latent warping strategy for keyframes and local frames, combined with token merging that uses a hybrid correspondence mechanism that integrates spatial information, optical flow, and feature-based matching. We show that our method not only achieves top performance in zero-shot video restoration but also significantly surpasses trained models in generalization across diverse datasets and extreme degradations (8$\times$ super-resolution and high-standard deviation video denoising). We present evidence through quantitative metrics and visual comparisons on various challenging datasets. Additionally, our technique works with any 2D restoration diffusion model, offering a versatile and powerful tool for video enhancement tasks without extensive retraining.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Video restoration Zero-shot Training-free Diffusion models
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/58b7cf74c09198469d992dfbe39811dba82077a2.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/ece9b098ba89dfe1e82311bd70fc3f703077883c.zip
DiffIR2VR-Zero: Zero-Shot Video Restoration with Diffusion-based Image Restoration Models
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission