SynCamMaster: Synchronizing Multi-Camera Video Generation from Diverse Viewpoints
Video Generation;Diffusion Model
Yes
Yes, Responsible research practice (e.g., human subjects, data release)
See Weaknesses.
The proposed method is simple yet effective for multi-camera video generation. 
The paper is well-written and easy to understand.
The results are impressive.
In this manuscript, the authors proposed a plug-and-play module that enhances a pretrained text-to-video model for multi-camera video generation. To achieve this goal, a multi-view synchronization module to ensure appearance and geometry consistency across different viewpoints. The synthesized videos using UE are generated as supplement for training. The proposed SynCamMaster is compared with related works on multi-view video generation, and provides competitive performance.
[Pioneer Status] In terms of pioneering work, it has come to my attention that Sora possesses multi-camera video generation capabilities. Therefore, the claim in the article that "To our knowledge, SynCamMaster pioneered multi-camera real-world video generation" warrants reconsideration.

[Novelty of SynCamMaster] The core design of SynCamMaster revolves around the inter-view synchronization module, which can be integrated into a pretrained text-to-video model. However, similar modules are commonly found in other generation models employing cross-view attention, which suggests that the novelty is incremental. It would be beneficial to emphasize how SynCamMaster distinguishes itself from existing methods.

[Camera Encoder] The process for converting 12-dimensional extrinsic parameters to align with the dimensions of spatial features remains unclear. The encoding of camera extrinsics significantly influences the final outcome, so providing additional crucial details would be appreciated.

[Typos] M.V. Images in table 1 and 4, they are mentioned in the manuscript. It is better to make it clear what is it. 

[Supplementary files] If the generated video files by all comparison methods are provided as supplementary files, it will provide a better and more intuitive way to understand the proposed SynCamMaster.
Yes
No ethics review needed.
My concern is mainly about the evaluation of multi-camera videos. Please refer to Weaknesses.
1. The method is novel and each module and design are reasonable with clear motivation.

2. The proposed method achieved good performance both qualitatively and quantitatively.

3. Extensive experiments and ablation studies demonstrate the effectiveness of the proposed techniques.
This paper focuses on the multi-view video generation with image/video and camera pose  conditions. Specifically, this paper propose a plug-and-play multi-view synchronization module to maintain appearance and geometry consistency. In addition, they collect a small set of 4D data and combine it together with 3D data to improve the generalization ability.
1. I am kind of confused with the motivation of this task. The multi-camera videos generated by this work are several videos with fixed cameras. However, the videos we see and use in our daily life are commonly one monocular video with a dynamic camera, like the task that MotionCtrl focuses on. I hope the authors could provide more explanation about the task motivation.

2. As claimed in the introduction section, multi-camera synchronized video generation is crucial for virtual filming. I am not an expert in virtual filming, but I feel it requires highly spatio-temporal consistent videos. Therefore, a good way to demonstrate the 4D consistency is to lift the generated samples to 4D representations, such as 4D gaussian splatting, and to show the rendered views. Plenoptic dataset [A] can be used for this evaluation.


3. Evaluation data and metrics should be explained. The details of the evaluation data are not provided. It is more convincing to evaluate on the public dataset. For multi-camera video evaluation, the PSNR, SSIM, and LPIPS metrics on Plenoptic [A]  D-NeRF [B] datasets can be used. For video quality evaluation, VBench [C] would be better than ``100 manually collected text prompts’’. In addition, when evaluating the accuracy of camera control with TansErr, the estimated camera should be normalized before estimating the loss (following [D]), since the SfM is up to a scale and the scale mismatch can greatly increase the TransErr. In CameraCtrl, the scale problem is not significant as the compared methods are all trained on the Re10K dataset. However, this paper trains on a different dataset (DL3DV-10K) with MotionCtrl (Re10K) and the scale problem might be enhanced.

*Reference*

[A] Neural 3D Video Synthesis from Multi-view Video

[B] D-NeRF: Neural Radiance Fields for Dynamic Scenes

[C] VBench: Comprehensive Benchmark Suite for Video Generative Models

[D] CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generation
Yes
No ethics review needed.
1. Around the line #189, should $\phi_t$ be $\psi_t$? And what do $\psi_t^{-1}$ and $\psi_t'$ mean, respectively?

2. Could you please provide more information about the real-world monocular videos used in this paper? Like duration, amount, how they are curated, etc.

3. In the line #140, does $f$ indicate the number of frames?

4. Is there any plan to release the code?

5. Can more visual examples of the generated multi-view synthetic videos be shown?
1. The multi-view video generation is a novel task. This paper shows some promising results in this direction, where the temporal consistency within each single view and spatial consistency across different views are maintained reasonably well.

2. The proposed way of using multi-view images, synthetic multi-view videos, and real-world monocular videos to train the model is novel. Ideally, we could train the model with multi-view videos, which are however hard to obtain, especially the high-quality real-world ones. The proposed approach shows an alternative approach to solve the data scarcity issue.
This paper presents a multi-view video generation approach based on an input textual prompt or monocular video. To achieve this goal, a multi-view synchronization module is incorporated into a pre-trained text-to-video model, where the goal is to maintain appearance and geometry consistency across multiple viewpoints. To address the scarcity issue of the training data, a progressive training scheme is designed that leverages multi-camera images, synthetic multi-view videos, and real-world monocular videos.
1. The proposed approach can only generate a single video at a novel viewpoint each time. Compared with generating multiple videos at different viewpoints simultaneously, the view consistency among the generated multi-view videos may not be good. Of course, the proposed approach can be used progressively generate further and further novel-view videos. But the view inconsistency may accumulate.

2. I wish the authors could show more visual results so the quality of the generated videos could be more comprehensively gauged. In the current results (both in the paper and on the project webpage), the same example (e.g., the gorilla eating carrot, the elephant walking on a beach) appears many times.
Yes
No ethics review needed.
See weaknesses.
1. Simplicity and Efficiency: The proposed method is both straightforward and efficient. This paper introduces a multi-view synchronization module designed to ensure camera-conditioned video consistency, enhancing realism and alignment across views.

2. Innovative Data Construction: This paper addresses the scarcity of multi-view video data by developing a robust hybrid-data collection pipeline. This process effectively collects diverse and comprehensive training data.

3. Thoughtful Training Strategy: The paper provides a clear breakdown of each stage in the training strategy, along with detailed insights into the motivations behind each stage. These insights offer readers a better understanding of the model's optimization process and rationale.

4. High-Quality Visualizations: The visualization results presented in the paper are impressive.

5. Clarity of Presentation: The paper is well-organized and articulated in a way that is accessible and easy to understand.
This paper introduces SynCamMaster, a pioneering framework for generating real-world videos across multiple camera perspectives. SynCamMaster also extends to novel view synthesis, allowing re-rendering of input videos from new viewpoints. To address the scarcity of multi-camera video data, the authors propose a hybrid data construction and training paradigm that enhances the model's generalization capabilities. Extensive experiments demonstrate that SynCamMaster significantly outperforms baseline methods, with particularly impressive visualization results.
Lack of Comparison with MotionCtrl: A drawback is the absence of a comparative evaluation with MotionCtrl ("MotionCtrl: A Unified and Flexible Motion Controller for Video Generation").
Yes
In the paper, authors generate some human (little girl) images through the diffusion model, I'm not sure whether it has ethics problem.
Yes, Privacy, security and safety
1.The citations for each method in tables should be added.
2.It appears a new synthetic dataset was proposed in this work for training the model. Will you decide to release this dataset? The dataset could be considered a contribution to the paper, so I suggest naming this synthetic dataset for readability.
3.In the qualitative experiments, you mention a polar bear case, but Figure 5 uses an elephant and a bus, and I couldn’t find polar bear images in the appendix.
4.In the quantitative experiments, your method performs worse than I2V-ours in terms of visual quality. I would like to see some analysis on this comparison between your results and those of I2V-ours.
5.Table 1 entries for I2V and SVD-XT should be unified if they refer to the same method for consistency.
6.In Figure 6(a), joint training strategy ablation study results are shown, but without ground truth to assess and with images that are too small, it is difficult to judge if the strategy improved the generation results, or if the effect is simply not significant. In contrast, Figure 6(b) shows clear improvements.
This paper introduces a forward-looking approach by achieving multi-camera real-world video generation, a pioneering advancement that is likely to drive significant progress in several areas in computer vision. Extensive experiments and ablation studies support the reliability of the proposed model, and the project website showcases video results that validate the method's effectiveness.
The paper "SynCamMaster: Synchronizing Multi-Camera Video Generation from Diverse Viewpoints" introduces a novel approach for generating synchronized multi-view videos from arbitrary viewpoints, addressing challenges in open-domain video generation that require consistent geometry and appearance across diverse camera angles. The authors leverage a pre-trained text-to-video model and enhance it with a multi-view synchronization module, designed to integrate inter-view feature consistency, which is achieved by incorporating a camera encoder and a unique attention mechanism. Additionally, they introduce a hybrid dataset and progressive training techniques to improve the generalization of the model to open-domain scenarios, effectively overcoming the scarcity of synchronized multi-view video data. Their experiments demonstrate superior cross-view synchronization and visual fidelity compared to baseline methods.
1.I believe the challenge identification section is unclear. Initially, the authors mention two challenges in this field, then state, "our research targets open-domain video generation from multiple arbitrary viewpoints, which has not been investigated." However, they do not clearly explain the reason for this research, nor do they clarify the limitations of existing methods and the higher demands that necessitate this research.
2.The explanation of the model structure and model formulation is severely lacking, making it difficult to connect the input-to-output process in the model with the text description, including the loss function and optimization mechanism. Specifically, the figure should indicate what the input is, which step produces the features, and how the loss is calculated, among other key details.
3.How does SynCamMaster determine the rotation center and radius when generating rotating camera viewpoints? I think that for a method designed to generate multi-view videos, this is an aspect that requires clarification.
4. Although the paper provides results for camera rotations like 90 degrees or 20 degrees, how can you quantitatively assess that the generated images match the rotation you specified? Without ground truth for reference, it is challenging to determine whether the rotation results are accurate. Could the authors use a COLMAP or similar method to register the spatial relationships of the generated multi-view images in camera space?
Yes
No ethics review needed.
Please refer to the weakness section.
(1) A multiview video dataset is presented.
(2) A multiview video diffusion model is trained using the proposed dataset.
(3) Comprehensive visualization results of the proposed method are provided.
This paper presents a multiview video dataset rendered using Unreal Engine. Additionally, it proposes a multiview video generative model architecture and training strategy to adapt a pretrained video diffusion model for simultaneous multiview generation.
(1) Encoding the camera pose matrix directly into the diffusion process as a condition is overly coarse. The authors should consider transforming camera poses into more explicit representations, such as ray positional encoding [1].
(2) Since the multi-view video is rendered rather than directly collected from real-world scenes, a significant concern is ensuring that the model learns only multi-view-related information rather than features specific to Unreal Engine-rendered models, materials, and textures. For instance, in Fig. 6 (a), how to avoid the output as the rendered scene? Authors should also introduce some real-world multiview videos to help balance the training of the diffusion model.
(3) The rotation angle in Fig. 6 (b) appears noticeably less than 90 degrees. Additionally, the precision of camera control is low, as indicated in Table 4.
(4) The proposed multi-view synchronization module is essentially a cross-attention layer, which lacks explicit modeling of 3D information.
(5) Given that camera poses are static in each rendered video under the proposed setup, it's unnecessary to render arbitrary camera poses. We can greatly lessen the burden of generative models and improve it accuracy by capturing and training more fewer camera poses, e.g., 6 views evenly distributed around 360 degrees, and applying 3D reconstruction on the generated views to get other different views.
[1] Gao, Ruiqi, et al. "Cat3d: Create anything in 3d with multi-view diffusion models." arXiv preprint arXiv:2405.10314 (2024).
Yes
Human faces are shown in the paper. Not sure if they are generated or sampled from some datasets.
Yes, Privacy, security and safety Yes, Responsible research practice (e.g., human subjects, data release)
1. Will the data preparation process and resulting dataset be made publicly available? The paper states that data was collected from a game engine, which could be valuable for advancing research in multi-view video generation. Public access to this data would enable reproducibility, facilitate benchmarking, and help other researchers develop and test models in a similar setting.

2. How were the data mixture probabilities (0.6, 0.2, 0.2) chosen, and what role do they play in the model’s performance? It would be helpful to understand the rationale behind this distribution and whether alternative ratios were explored. Insights into how these probabilities impact training dynamics and model generalization would also clarify the importance of the data mixture balance in achieving optimal results.
1. The approach of generating synchronized multi-view videos is a strong point, as these videos offer applications beyond traditional 4D reconstruction, such as enhancing virtual filming. By maintaining dynamic consistency across views, this method broadens the potential use cases in various real-time and interactive settings.

2. The visual quality of the generated videos is impressive, particularly with respect to the wide camera baselines achieved. This indicates the model’s robustness in handling significant viewpoint shifts, preserving both appearance and geometric consistency across diverse perspectives, which is essential for applications requiring high-fidelity visual continuity.

3. Leveraging multi-view video data from game engines for fine-tuning is a smart approach, especially given the scarcity of high-quality multi-camera real-world datasets. The method demonstrates effective generalization to real-world data, showing that game engine data can be a viable supplement for improving model performance in open-world video generation tasks.
This paper introduces a plug-and-play module for generating consistent videos from arbitrary viewpoints using 6 DoF camera poses. Unlike methods focused on 4D reconstruction of single objects, this approach enhances a pre-trained text-to-video model for synchronized multi-camera generation, ensuring appearance and geometry consistency across viewpoints. To overcome limited training data, this paper uses a progressive training scheme, leveraging multi-camera and monocular videos along with Unreal Engine-generated data.
1. Although 4D video reconstruction is highlighted as a key application, the paper does not provide specific results on this. Including an evaluation of 4D reconstruction performance (like 4DGS) on the output synchronized videos would strengthen the contribution and showcase the practical utility of the generated multi-view videos in real-world 4D reconstruction tasks.

2. The use of camera conditioning is potentially limited, as it may not handle large out-of-distribution camera pose translations during inference (e.g., for training only 3.5m - 9m shifts). Testing the model on such out-of-distribution poses would provide valuable insight into its robustness and adaptability to extreme viewpoint variations, which are common in real-world applications.

3. Multi-view consistency could be better assessed. A qualitative evaluation, such as a cycle consistency check, could help—rendering videos from generated camera poses and then re-rendering from the original poses would allow comparison. The differences between these two renderings could serve as a metric for the model’s ability to maintain consistent content across varying views, giving a clearer measure of its multi-view coherence.
A efficient method is proposed to lift pre-trained text-to-video models for open-domain multi-camera video generation from diverse viewpoints
@inproceedings{
anonymous2024syncammaster,
title={SynCamMaster: Synchronizing Multi-Camera Video Generation from Diverse Viewpoints},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=m8Rk3HLGFx},
note={under review}
}
Recent advancements in video diffusion models demonstrate remarkable capabilities in simulating real-world dynamics and 3D consistency. This progress motivates us to explore the potential of these models to maintain dynamic consistency across diverse viewpoints, a feature highly sought after in applications like virtual filming. Unlike existing methods focused on multi-view generation of single objects for 4D reconstruction, our interest lies in generating open-world videos from arbitrary viewpoints, incorporating six degrees of freedom (6 DoF) camera poses.
To achieve this, we propose a plug-and-play module that enhances a pre-trained text-to-video model for multi-camera video generation, ensuring consistent content across different viewpoints. Specifically, we introduce a multi-view synchronization module designed to maintain appearance and geometry consistency across these viewpoints. Given the scarcity of high-quality training data, we also propose a progressive training scheme that leverages multi-camera images and monocular videos as a supplement to Unreal Engine-rendered multi-camera videos. This comprehensive approach significantly benefits our model.
Experimental results demonstrate the superiority of our proposed method over existing competitors and several baselines. Furthermore, our method enables intriguing extensions, such as re-rendering a video from multiple novel viewpoints.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Video Generation Diffusion Model
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/03ce71fd6c73b735e5d151442d0c7aab56c4e46a.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
SynCamMaster: Synchronizing Multi-Camera Video Generation from Diverse Viewpoints
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission