MVDrag3D: Drag-based Creative 3D Editing via Multi-view Generation-Reconstruction Prior
3D Editing;3D Drag;Multiview Diffusion Model
Yes
No ethics review needed.
- Why do the authors regress a partial 3D Gaussians for each view and fuse them afterwards instead of performing a multi-view reconstruction?
- Referring to Table 1, is there any explanations why SDS optimization produces better results for Gaussians than for meshes in terms of DAI?
- For 3D meshes, are the final outputs still 3D Gaussians? Are the quantitative results all computed on 3D Gaussians?
+ The idea of casting 3D drag-based editing into multi-view 2D drag-based editing sounds novel and feasible.
+ The use of multi-vew diffusion model to ensure consistent multi-view image editing sounds novel and feasible.
+ The lighweight view-specific deformation networks appear to be effective in improving the geometric alignment of the 3D Gaussians with the image.
+ The use of image-conditioned multi-view SDS optimzation for enhancinmg view consistency and visual quality sounds logical.
+ Both qualitative and quantitative results demonstrate SOTA results.
In this paper, the authors present a framework named MVDrag3D for drag-based 3D editing. It first renders a 3D object and the drag points into 4 orthogonal views, and introduce a multi-view guidance energy to achieve consistent multi-view score-based image editing. Multi-view Gaussian reconstruction is then performed on the edited images, followed by Gaussian position adjustment by view-specific lightweight deformation networks. Finally, an image-conditioned multi-view SDS optimization is applied to further enhance view consistency and visual quality.
- There is no disucssions on how to choose the optimal 4 orthogonal views. Theoretically, the 4 orthogonal should be chosen such that the drag directions should be as far away from the view directions as possible. In this paper, the authors simply choose orthogonal azimuths (0 deg, 90 deg, 180 deg, 270 deg) and a fixed elevation (0 deg).
- By rendering only 4 orthogonal views, some details (shape and texture) of the 3D object may be lost.
- In 3D Gaussian reconstruction step, partial 3D Gaussians are regressed for each view, which are then fused into a unifed representation. It is not clear why the authors do not perform multi-view Gaussian reconstruction instead.
- It is also not clear wheather the Gaussian position optimization step is necessary. Why can't the image-conditioned multi-view SDS optimization be carried out directly on the initial 3D Gaussians?
Yes
No ethics review needed.
-- Does the proposed method ultimately produce a 3DGS representation for editing the input mesh? What if we still want a mesh-based output? While 3DGS can extract the mesh, the quality of the extracted mesh is not very good, and it is necessary to use higher-quality representations like 2DGS for better reconstruction results.

-- While the deformation MLP and multi-view SDS loss optimization do have some effectiveness in fine-tuning the final results, they seem to be suitable primarily for relatively minor issues. For more significant geometric inconsistencies and texture artifacts, it remains unclear whether they can adequately address these challenges. And could the video include results from ablation experiments related to these two aspects?
-- The paper has a clear description and is well-structured, providing not only the necessary details for implementing the method but also the motivation for choosing this technical approach.

-- The method proposed in this paper is reasonable and has achieved results superior to other comparative methods.
This paper proposes a 3D editing method based on sparse control point dragging, applicable to both neural implicit representations and explicit mesh representation. The proposed method is based on a multi-view diffusion model and a large model for sparse view reconstruction. Specifically, the method renders the 3D representation into images from four specified viewpoints, then uses DDIM inversion to convert these images into Gaussian noise, which is regenerated using the multi-view diffusion model MVDream. During the generation process, control point movements are incorporated as a constraint to define multi-view guided energy optimization for intermediate denoising features. The generated edited images from the four viewpoints will obtain a 3D GS representation using the large pre-trained model, LGM. However, the 3D Gaussians obtained from the four views exhibit geometric deviations and texture artifacts. The former is addressed by introducing an MLP to shift the positions of Gaussian spheres, while the latter is resolved through multi-view SDS loss optimization. The paper compares the proposed method with other similar drag-based editing methods.
-- The method proposed in this paper is straightforward but lacks sufficient technical contribution. The first step generates multi-view edited images, with its optimization method and energy terms resembling those of Dragondiffusion. The main difference is merely substituting a single-view diffusion model with a multi-view one, which is a simple and direct change requiring minimal adjustments. In the second step, the LGM is used to regenerate the 3DGS model by directly utilizing an existing pre-trained model. Finally, the adjustments made through the deformation MLP and multi-view SDS loss optimization are common practices in 3D modeling and generation. Therefore, the proposed method primarily combines existing approaches without significant improvements, leaning more towards engineering rather than showcasing technical contributions.

-- Although the proposed method outperforms existing approaches in terms of deformation effects, it is still limited by the 3DGS representation, resulting in some artifacts and blurriness at the model edges in the final output. For instance, this is evident in the green leaves of the flowers, the open mouth of the crocodile, and the open mouth of the lion in the video.
Yes
No ethics review needed.
* Listed in the wakenesses.
* MVDrag3D is capable of user-friendly drag operations. Extending the idea of 2D drag to 3D makes sense and shows good qualitative results.
* MVDrag3D shows the ability to generate new textures, which makes it different from the standard 3D deformation works.
This paper proposes MVDrag3D, which utilizes a deformation network and SDS loss from multi-view diffusion models to edit 3D objects. The results show that the proposed method can generate 3D creations with the corresponding drag operation while creating new textures.
* Low-quality 3D creations: Most of the results are based on simple 3D creations or generated 3D creations, which are pretty low-quality. For example, the shoes and the fox show a blurred texture. Such evaluation dramatically limits the application of the proposed method. In addition, the edited results in the video tend to be blurred compared with the "Drag on meshes" illustration. These observations raise two concerns: (1) will this method blur the original 3D content? (2) will this method work for high-quality 3D creations?
* Failure cases of drag operations: The drag operation does not usually seem successful in the examples. The drag on the owl is not correctly optimized since the cloth doesn't reach the target position. In addition, the texture of the tie is degenerated during this process. Such a result even sees worth than the baselines like APAP. The use of SDS loss probably hurt the results to some extent.
Yes
N/A
No ethics review needed.
I have raised questions in the weakness section
The paper is a system paper but t does has some interesting novel ideas that might be beneficial to the research community. There are several notable strengths
- The proposed approach is sensible in that leverating multi-view to control the dragging behavior is reasonable, and the overall formulation of the algorithm is intuitive.
- The introduced multi-view guidance score and the refinement approach is novel in that it helps reduce artifacts and addressed softspots in the algorithm. There are several mini-ideas in the approach that are interesting, such as adding random noises in the inversion, etc.
- Experimental results looks competitive, especially some of the visual comparisons.
The paper proposed a new algorithm called MVDrag3D, which is a framework for drag-based 3D editing. The method first generates multi-view images, followed by mechanisms introduced to enforce consistencies in the generation process, before reconstructing a 3D model. In the experiments, it looks like the proposed approach provides a more precise 3D reconstructed results comparing to other state of the art approaches
I think there are merits in the paper but i've got some questions or confusions while reading the paper
- In figure 3, the differences of the final reconstruction results are very minor to me. I couldn't fully appreciate the need of the random noise unless I zoom into the pictures. Technically none of those results matches the input very well.
- Experimental results are conducted on vey simple 3D objects. Maybe this is a common issue for all state of the art approaches. Similarly, in fig. 6 -- many appraoches perform quite competitively (such as the boots, flower, and the suited animal case), and I struggle to tell if the approach is truly groundbreaking or incremental
- The descriptions of the technical formulations of the approach requires more details. For example, i struggled to understand the "Gaussian appearnace optimization" section -- "Note that all Gaussian properties are optimized during this process, with densification and pruning operations enabled." -- which densification and pruning approach exactly, and how is this done? The "shoe" example in fig.5 is also showing tiny improvements with the optimizations.
- It'd be good if the paper could provide human evaluations. The automatically computed metrics might not tell the full story of the effectiveness of the approach.
@inproceedings{
anonymous2024mvdragd,
title={{MVD}rag3D: Drag-based Creative 3D Editing via Multi-view Generation-Reconstruction Prior},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=igpiMCYEjM},
note={under review}
}
Drag-based editing has become popular in 2D content creation, driven by the capabilities of image generative models. However, extending this technique to 3D remains a challenge.  Existing 3D drag-based editing methods, whether employing explicit spatial transformations or relying on implicit latent optimization within limited-capacity 3D generative models, fall short in handling significant topology changes or generating new textures across diverse object categories. To overcome these limitations, we introduce MVDrag3D, a novel framework for more flexible and creative drag-based 3D editing that leverages multi-view generation and reconstruction priors.
At the core of our approach is the usage of a multi-view diffusion model as a strong generative prior to perform consistent drag editing over multiple rendered views, which is followed by a reconstruction model that reconstructs 3D Gaussians of the edited object. While the initial 3D Gaussians may suffer from misalignment between different views, we address this via view-specific deformation networks that adjust the position of Gaussians to be well aligned. In addition, we propose a multi-view score function that distills generative priors from multiple views to further enhance the view consistency and visual quality. Extensive experiments demonstrate that MVDrag3D provides a precise, generative, and flexible solution for 3D drag-based editing, supporting more versatile editing effects across various object categories and 3D representations.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
3D Editing 3D Drag Multiview Diffusion Model
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/26a8ebec700948e6c321fa707d7fa9ae8afaf0b1.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/e6b144818938c27ac3c99d8cc8f88e052e542a8b.zip
MVDrag3D: Drag-based Creative 3D Editing via Multi-view Generation-Reconstruction Prior
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission