MoDGS: Dynamic Gaussian Splatting from Casually-captured Monocular Videos
3D Gaussian Splatting;Dynamic Novel-view Synthesis;Neural Rendering
Yes
No ethics review needed.
From the perspective of a peer, I suggest the authors address the concept of 'Depth supervised in dynamic GS' in the title. After all, a novel method would be more informative and important to the other researchers than a usage scenario like ''casually captured video".
For the novelty, this paper makes a distinct contribution to introducing depth supervision into the domain of dynamic Gaussian Splatting (DGS) for monocular dynamic input. This approach is novel yet intuitive, filling a key gap in the field for cases where the input consists of casually captured videos with minimal camera movement. Compared to the other papers in the field that mechanically put all fancy complicated input feature streams or loss functions together, the proposed solution is conceptually straightforward but impactful, pushing forward the capabilities of monocular dynamic scene reconstruction.

The experiments are well-designed and executed, rigorously testing the proposed method across various datasets, including Nvidia, DyNeRF, and DAVIS. Each experiment logically supports the methodology, demonstrating how the 3D-aware initialization and ordinal depth loss contribute to enhanced depth consistency and scene deformation modeling. The results clearly show MoDGS’s robustness and superiority over baseline methods, adding confidence in its effectiveness.

The paper is presented with clarity and precision, making even technical aspects of the method easy to follow. The figures and tables are well-constructed and informative, providing visual clarity to support the text and helping to reinforce the main findings. The logical flow, from problem statement to method explanation and results, enables readers to understand the method's motivation and benefits seamlessly.
The paper introduces MoDGS (Monocular Dynamic Gaussian Splatting), a novel approach for rendering dynamic 3D scenes from casually captured monocular videos, overcoming limitations faced by prior dynamic NeRF and Gaussian Splatting methods via depth estimation. These existing approaches require either extensive camera movement or synchronized multi-view setups to establish multiview consistency, which is lacking in casual, minimally moving videos.

To tackle this challenge, MoDGS incorporates recent advancements in single-view depth estimation to guide the learning of a deformation field that represents scene dynamics. The method introduces a novel ordinal depth loss to address the depth inconsistency in single-view depth maps, enhancing the robustness and continuity of 3D scene reconstruction.

Comprehensive experiments across multiple datasets (Nvidia, DyNeRF, DAVIS, and a self-collected casual video dataset) demonstrate that MoDGS produces high-quality novel views in dynamic scenes, outperforming state-of-the-art methods. The authors also plan to release their code and dataset to support future research in this area.
MoDGS is validated across several datasets, which demonstrates its robustness. However, the paper could discuss the potential limitations in generalizing this approach to different depth estimation models. It would demonstrate the robustness of the proposed method and its generalizability.
Yes
A version of this paper is available on arxiv https://arxiv.org/pdf/2406.00434, and I had viewed a tweet earlier in the summer with the same title, paper, code: https://x.com/zhenjun_zhao/status/1798281777242632700. This may violate the double-blind review that is required, so I would like that to be known.
Yes, Other reasons (please specify below)
1. Can you elaborate on the choice of ordinal depth loss over other depth loss functions, such as perceptual depth consistency? How did the ordinal depth loss compare to other depth loss formulations in preliminary experiments, and what were the observed advantages or disadvantages?
2. How robust is MoDGS in scenarios with heavy occlusions or specular reflections? Would integrating additional priors or multi-scale depth estimations help in such cases?
3.   How does MoDGS compare with recent depth consistency techniques, particularly those used in self-supervised monocular depth estimation? Exploring this comparison could shed light on the effectiveness of the ordinal depth loss relative to existing methods.
MoDGS represents an original approach within  novel view synthesis  and dynamic scene modeling by specifically addressing the limitations of existing methods for casually captured monocular videos. The authors introduce a 3D-aware initialization mechanism and an ordinal depth loss, that offer a solution that successfully reduces the dependency on rapid camera motion. The novel use of ordinal depth loss to maintain depth order among frames, rather than relying solely on absolute values, represents an innovative perspective on addressing depth consistency issues, which has practical implications for improving depth coherence in dynamic scenes captured casually. I believe the paper is well-executed in terms of technical rigor, with comprehensive evaluations across three datasets: DyNeRF, Nvidia, and a newly created monocular casual video dataset. Each component of MoDGS is thoroughly tested and ablated to demonstrate its impact on the final results. This systematic experimentation supports the author’s claim that MoDGS significantly outperforms other approaches in the quality of novel-view rendering for dynamic scenes. The paper is structured logically, with clear explanations of each component of the MoDGS pipeline. The figures visually support the textual explanations, making complex concepts more understandable to a reader. The method  has significant implications for real-world applications that involve casually captured videos, such as mobile AR/VR, video editing, and 3D content creation. By enabling high-quality novel view synthesis from single-camera footage without multiview camera motions, MoDGS broadens the scope of dynamic scene reconstruction, making it accessible to a wider range of use cases. The method’s ability to handle both static and dynamic elements in monocular videos opens new avenues for monocular depth estimation and dynamic scene modeling, where single-camera approaches have been historically constrained by depth inconsistency issues.
The paper proposes MoDGS, a novel pipeline to render high quality novel views of dynamic scenes from casually captured monocular videos. Unlike traditional dynamic scene reconstruction methods that rely on rapid camera motions to establish multiview consistency, MoDGS is designed for videos with static/slowly moving cameras, where such consistency is weaker. The core of their method involves using a single-view depth estimation technique to guide scene learning and introducing a 3D-aware initialization method to construct a realistic deformation field. MoDGS incorporates an innovative ordinal depth loss to address the challenge of depth inconsistency across frames, enhancing the coherence and quality of rendered views. Experiments on datasets such as DyNeRF, Nvidia, and a self-collected dataset demonstrate it's ability to outperform SOTA methods in novel view synthesis, achieving superior image quality even in challenging dynamic scenarios.
While the ordinal depth loss is a novel way to improve depth coherence, I believe the paper may benefit from more discussion on its limitations. Specifically, the ordinal depth loss assumes a consistent depth order among frames, which may not hold in scenes with complex occlusions or reflections. MoDGS assumes smooth transitions between frames for consistent depth ordering. However, the approach may face challenges in scenes with rapid or erratic movement where objects appear and disappear frequently. While it performs well on scenes with relatively smooth dynamics, addressing how the method might be adapted or optimized for highly dynamic environments would improve its versatility. The method relies heavily on single view depth estimators to guide the reconstruction process. Although the depth estimation technique used is SOTA, it still inherits the limitations of single view estimators, particularly in complex scenes with specular surfaces or low-lit conditions. Including a more detailed analysis on how the quality of the depth estimator impacts the proposed method’s performance, and potentially exploring integration with other depth supervision methods could potentially make the approach more adaptable across varying input qualities.
Yes
No ethics review needed.
Kindly refer to the [Weaknesses].
* A differentiable order-based loss function, the ordinal depth loss, is proposed, with detailed descriptions of its motivation and its distinctions from other depth loss functions.
* It demonstrates significant superiority over multi-view camera methods in reconstruction metrics and visual results, with ablation studies validating the importance of the "3D-aware initialization scheme" and "ordinal depth loss."
* The paper is well-written and easy to follow.
The paper presents MoDGS, a novel pipeline for synthesizing dynamic scenes from casually captured monocular videos. Unlike existing methods requiring large camera motions, MoDGS leverages single-view depth estimation for 3D reconstruction and introduces a 3D-aware initialization alongside an ordinal depth loss. These innovations enable robust, high-quality novel view synthesis, outperforming state-of-the-art methods in rendering casually captured videos.
* **The contributions and innovations are limited**. This work is based on the previous canonical space paradigm of 3D Gaussian Splatting (3DGS) combined with deformation fields, with the main contributions being a deformable 3DGS initialization method and a depth loss. The primary principle of the former relies on predicting per-pixel 3D flow using current state-of-the-art monocular depth estimation and optical flow estimation methods. However, the sole innovative aspect lies in converting 2D optical flow to 3D flow using the estimated depth map. As for the depth loss, although it is well-motivated and provides performance improvement, it essentially replaces the Pearson correlation loss with an order correlation loss.
* **The experimental comparisons lack fairness**. In most quantitative comparisons, this work is only compared against methods that require multi-view camera input. It is recommended to include quantitative and qualitative comparison results with methods under the same setting of "casually captured monocular video." It is also perplexing that the authors mention "RoDynRF, a method that adopts single-view depth estimation as supervision" in "Baseline methods", yet I only found comparative results for this method in Fig. 6.
Yes
No ethics review needed.
1. How does MoDGS handle scenarios where the pre-trained depth estimator provides inconsistent depth due to environmental variations? Has any analysis been conducted to measure performance stability when GeoWizard or other models are less reliable?  


2. Would MoDGS perform as well on datasets with higher motion complexity or less predictable scene geometry? Testing on a broader range of datasets, such as those with cluttered backgrounds or multiple moving objects, would better validate the method's generalization.   

3.  Considering MoDGS’s reliance on single-view depth priors, would a formalized knowledge distillation framework improve model autonomy by adapting these priors dynamically during training?
1. I think the 3D-aware initialization process is a strong point, as it specifically addresses a common issue in monocular reconstruction. By initializing Gaussians instead of relying on random initialization, this method seems to potentially add more consistency.  

2. The ordinal depth loss is, in my view, an interesting idea. It tries to tackle scale ambiguity in monocular depth estimation, which I think is particularly relevant in dynamic scenes. This loss formulation promotes depth consistency across frames, an essential factor when handling complex, moving scenes.
The paper introduces MoDGS, a method for dynamic view synthesis from monocular videos, leveraging a Gaussian-based splatting technique combined with deformation fields and an ordinal depth loss to reconstruct scenes. This framework integrates a 3D-aware initialization to align Gaussian representations in a canonical space, while the ordinal depth loss is used to improve scene geometry continuity. MoDGS is claimed to improve over previous dynamic NeRF approaches and related deformation methods, with results evaluated on the DyNeRF and Nvidia datasets.
1. I think the innovation is quite incremental for the reason that compared to closely related works like Deformable 3DGS and 4DGS, the methodological innovation appears incremental, mainly optimizing existing elements (depth consistency and deformation) rather than proposing a new structural approach.


2. Besides, the approach relies heavily on pre-trained depth models. MoDGS relies on single-view depth estimators like GeoWizard for depth initialization, which brings into question the independence of its results. The approach leverages external models as priors, potentially limiting its novelty and raising questions regarding knowledge distillation. The extent to which these pre-trained models influence the final performance is not rigorously analyzed.  
 

3. While MoDGS integrates external depth estimation for initialization, there is no formalized knowledge distillation to adaptively refine the model during training. This absence may reduce the adaptability of MoDGS across different dynamic scenes where pre-trained depth estimators may not perform equally well.
@inproceedings{
anonymous2024modgs,
title={Mo{DGS}: Dynamic Gaussian Splatting from Casually-captured Monocular Videos},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=2prShxdLkX},
note={under review}
}
In this paper, we propose MoDGS, a new pipeline to render novel-view images in dynamic scenes using only casually captured monocular videos. Previous monocular dynamic NeRF or Gaussian Splatting methods strongly rely on the rapid movement of input cameras to construct multiview consistency but fail to reconstruct dynamic scenes on casually captured input videos whose cameras are static or move slowly. To address this challenging task, MoDGS adopts recent single-view depth estimation methods to guide the learning of the dynamic scene. Then, a novel 3D-aware initialization method is proposed to learn a reasonable deformation field and a new robust depth loss is proposed to guide the learning of dynamic scene geometry. Comprehensive experiments demonstrate that MoDGS is able to render high-quality novel view images of dynamic scenes from just a casually captured monocular video, which outperforms baseline methods by a significant margin.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
3D Gaussian Splatting Dynamic Novel-view Synthesis Neural Rendering
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/d6d3f312042a0e87b6a180ad4c5d66669ae79bbc.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/5a75f66168ff65f33d2bb5408418c355a24e2e5f.zip
MoDGS: Dynamic Gaussian Splatting from Casually-captured Monocular Videos
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission