VideoPanda: Video Panoramic Diffusion With Multi-view Attention
video generation;diffusion model;panorama
Yes
No ethics review needed.
1. Can VideoPanda generate videos conditioned on different FOVs and view directions?
2. What is the specific implementation of the multi-view attention, and how does it differ from existing methods?
3. Can the authors provide more quantitative and qualitative results without watermarks?
The concept of expanding single-view videos into panoramic videos is interesting. Some of the generated results appear visually good.
This paper introduces a method named VideoPanda, designed to synthesize 360-degree videos from text prompts or single-view videos. VideoPanda builds on existing video diffusion models by adding multi-view attention layers to produce consistent multi-view outputs. Both quantitative and qualitative results are presented.
1. The model appears to be overfitted to the WEB360 dataset. The test videos are directly sourced from the WEB360 dataset or selected from similar scenes on the airpano channel (e.g., the pandas in "100329" from the WEB360 dataset, and the ice and mountains in "100666"). The results generated by the method often contain the airpano watermark, whereas the few results without watermarks on the webpage exhibit strange artifacts in other regions. This indicates a lack of generalization to real-world scenarios.
2. The authors state, "Since the out-of-distribution condition inputs do not originate from 360 videos, we cannot compute metrics that require ground truth images, such as pairwise FVD." However, to the best of my knowledge, FID and FVD do not necessarily require paired prediction-ground truth data. FID and FVD are distribution-matching metrics and do not require one-to-one correspondence between generated and real data.
3. There are no ablation studies on the multi-view attention mechanism. The paper does not clearly explain the differences between the proposed multi-view attention and existing methods, such as MVDream.
4. The paper lacks experiments on conditions with varying fields of view (FOV) and view directions. The results only demonstrate conditioning using a 0-degree latitude image. In practical scenarios, adapting to different FOVs and viewing angles is a common requirement.
Yes
No ethics review needed.
- How many samples are used for the evaluation in Table 2? How do you collected the prompts? Do they cover good diversity?

- It is hard to understand why multi-task training is better than single task training. This is contrast with some common feeling, for example all the T2V or I2V modes are processed with fixed frames, because varied number of tokens in the attention computation may hinder the performance on a specific frames and resolution. I would like to see more elaboration over this.

- The artifacts of full matrix shown in Figure. 7, actually also happen in the result of the proposed method (i.e. random matrix), such as in the [video-conditioned generation] showcase "A view of a panda's face peeking from behind a tree branch amidst lush green foliage“.
So it seems both setting suffers from the over blurry and structure mixture artifacts.

- According to the paper, the model is finetuned with the base model layers frozen. In my understanding, the distribution of each view of the panorama still deviate with the original real-world image. Would it be helpful to the generation quality (such as the ambiguous/broken scene semantics) if the base model is tuned with LORA?
- This paper is well written and easy to follow. Every design is well-motivated and clear clarified.

- The key idea to formulate panorama video generation as multi-view video generation makes sense and novel. Quantiative results (i.e. Table 1) evidence the superiority over existing baseline method 360DVD.

- The experimental evaluation is well-conducted and sufficient. Multiple metrics are adopted and user study is performed.
This paper presents a novel method for long panoramic video generation from a text prompt or a perspective video. Different from existing 360-DVD that generates equi-rectangular panorama video directly, it builds on existing video diffusion models by adding multi-view attention layers to generate consistent multi-view outputs. It is expect to help maintain the capability of the pre-trained model without domain gap hindering. Extensive experimental results tells the superiority of the proposed method, especially the quantitative evaluation and user study. Anyhow, it still suffers from obvious visual artifacts, as shown in the demonstrated video results.
-The major weakness is that most of the generated panorama videos are not good as expected, which renders the importance of the key technical innovation not well supported.
First, most of the results present ambiguous semanic structure or broken scene, such as the [autoregressive generation] showcase "anime girl standing on a boat", and the [video-conditioned with different prompts] showcase "A view of the seashore with the sea coming up against a cliff.“ This is a unignorable weakness about the performance.
Besides, almost all results present obvious seamline across views.
It seems the newly introduced multi-view attention does not work as expected. A possible attempt is jointly finetuning the base model with LORA, which may help the model better adapt to the panorama video distribution.

-As stated above, some of results suffers from broken content and semantic ambiguity. In contrast, although showing over-smooth textures, the semantics of 360DVD are more natural and the scene of content are more identifiable than the proposed method. This impedes the justification of the proposed method is better than existing 360DVD.

-The comparative examples with 360DVD are almost static scene, which makes the evaluation less convincing. It is highly required to evaluate on cases with moving objects (such as "moving car on the street", "astronaut riding a horse on grass", etc.), because the consistency of dynamic objects is one of the major focus in video generation task.
Yes
No ethics review needed.
See weakness.
I will check the author's response and revise the score after combining other review comments.
1. The ideas are very easy to understand, reliable and well written.

2. The boost in quantitative metrics looks good.

3. Open source code for easy reproduction by readers
This work present a method for generating videos from text or single-view video data, which employs multi-view attention to enhance a video diffusion model and produce consistent multi-view content suitable for immersive panoramas. The work demonstrates performance improvements and provides code.
1.Poor qualitative results. I feel that the overfitting is evident in the effect, that is, the watermarks are being generated, and on closer inspection you can see the airpano.

2.Some of the diagrams in the paper don't have good consistency when zoomed in.

3.The technology is low-innovative, and multi-view concerns are common in the 3D AIGC field[1].

[1] Liu J, Huang X, Huang T, et al. A comprehensive survey on 3D content generation[J]. arXiv preprint arXiv:2402.01166, 2024.
Yes
No ethics review needed.
1. In Table 3, after applying the multi-task training strategy, several metrics, such as FVD and PSNR, show a decline. Could the authors provide an appropriate explanation for this?
2. In section 4.6, the authors tested autoregressive long video generation, producing videos up to 61 frames. Have the authors attempted to generate even longer videos? As video length increases, does generation quality continuously degrade? If so, what feasible strategies might help mitigate this issue?
3. In Table 1, the authors compare their method with 360DVD and report performance improvements. However, 360DVD uses the text-to-image model SD1.5, while the proposed method uses the SVD video generation model, which inherently offers an advantage in video smoothness. Did the authors conduct a relatively fair comparison, such as applying SVD to 360DVD or using SD1.5 with the proposed method?
1. The authors successfully adapted a pretrained video generation model to the multi-view/panoramic video generation task, achieving relatively good visual results.
2. The proposed method supports using text or video as conditions to generate panoramic videos.
This paper proposes a framework for generating panoramic videos, based on a pretrained video generation model. The authors introduce multi-view generation capabilities by embedding several Multi-view Blocks. Additionally, they propose a random matrix training strategy, using videos of random frames and views to train the model, which increases the model's generalization to longer/more viewed videos while conserving computational resources. The authors trained the model using WEB360 and tested their method on 100 in-distribution and 100 out-of-distribution videos.
1. In terms of model design, the authors introduced a Multi-view Block on top of SVD (Stable Video Diffusion) to enable multi-view generation, a concept similar to MVDiffusion. However, while MVDiffusion focuses on panoramic image generation, this paper is designed for panoramic video generation, adding a temporal block (see Figure 2). Essentially, this paper can be seen as an application combining SVD and MVDiffusion, with the spatial and temporal blocks derived from SVD and the multi-view block from MVDiffusion. From this perspective, the novelty of the proposed method may be somewhat limited, so it would be beneficial for the authors to explain how their approach differs from these methods and whether they have made specific design choices to address challenges unique to this application.
2. Regarding the training strategy, the proposed random matrix strategy is essentially a compromise due to limited computational resources; theoretically, using more views or frames in training would yield better results. From the experimental results in Table 3, it can be seen that improvements in FID and FVD scores are achieved at the cost of PSNR (rows 2 and 3).
As for the multi-task strategy proposed by the authors—randomly dropping some conditions (such as text, the first frame, or single-view video), or conditioning on only part of the information or a subset of modalities—is a common trick in training diffusion models. For example, dropping text is often used in text-to-image/video tasks [GLIDE, SVD, Imagen, etc.], or conditioning on text and image in image editing [InstructPix2Pix, Tim Brooks et al. 2022]. Therefore, it would be helpful for the authors to clarify how their approach differs from these methods to demonstrate the novelty of their method.
We enable text and video conditional generation of panorama videos using multiview video models.
@inproceedings{
anonymous2024videopanda,
title={VideoPanda: Video Panoramic Diffusion With Multi-view Attention},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=CMj18BQQDK},
note={under review}
}
High resolution panoramic video content is paramount for immersive experiences in Virtual Reality, but is non-trivial to collect as it requires specialized equipment and intricate camera setups. 
In this work, we introduce \ourmodel, a novel approach for synthesizing $360^\circ$ videos conditioned on text or single-view video data. \ourmodel leverages multi-view attention layers to augment a video diffusion model, enabling it to generate consistent multi-view videos that can be combined into immersive panoramic content. \ourmodel is trained jointly using two conditions: text-only and single-view video, and supports autoregressive generation of long-videos. 
To overcome the computational burden of multi-view video generation, we randomly subsample the duration and camera views used during training and show that the model is able to gracefully generalize  to generating more frames during inference.
Extensive evaluations on both real-world and synthetic video datasets demonstrate that \ourmodel generates more realistic and coherent $360^\circ$ panoramas across all input conditions compared to existing methods.
Visit the project website at https://mvpanovideo.github.io/VideoPanda/ for results.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
video generation diffusion model panorama
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/d3d85662b2e67875595b94d24ce5681c1c729c2c.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/39d2b08c7828294d4e83eb11e3ccf02f3d3d4eb4.zip
VideoPanda: Video Panoramic Diffusion With Multi-view Attention
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission