ZERO-1-to-G: Taming Pretrained 2D Diffusion Models for Direct 3D Generation
native 3D diffusion;3D generative model;gaussian splats
Yes
No ethics review needed.
Is it possible to use the single-image format of Splatter Image? In this way, the whole 3D scene can be encoded in one image-like tensor, thus avoiding the difficulty of constraining multi-view consistency. Of course, the VAE component also need to be retrained to adapt to the image-like tensor.
1. The idea of leveraging 2D diffusion priors for 3D generation is promising, and the adaptation of Splatter Image structure is technically sound.
2. A series of correspondingly needed modifications are applied. For example, the different Gaussian attribute maps are normalized/reorganized to RGB space, and some self-attention operations are added for the modeling of both intra-view and inter-view distributions.
This paper aims to leverage 2D priors in pre-trained image diffusion models for 3D content generation. The key idea is to adapt the previous work of Splatter Image as a proxy 2D representation of 3D information, such that multi-view image diffusion architectures can be applied.
1. Generally, I tend to think that the proposed processing pipeline is a relatively simple and straightforward combination of existing techniques. First, the concept of Splatter Image is an existing work, and in this paper the adaptations are limited to some normalization and channel duplication operations. Second, the attention-related operations are also well-explored in previous methods, such as the "reshaping for attention" in MVDream and the "switcher mechanism" in Wonder3D.

2. I disagree with the authors' claim that "Zero-1-to-G is the first direct 3D generative model based on 2D diffusion frameworks". In terms of conceptual novelty and the target of leveraging 2D diffusion architectures, Omage [R1] and GIMDiffusion [R2] are even more representative works. 
--[R1] X. Yan, et al., “An Object is Worth 64x64 Pixels: Generating 3D Object via Image Diffusion,” 2024.
--[R2] S. Elizarov, et al., “Geometry Image Diffusion: Fast and Data-Efficient Text-to-3D with Image-Based Surface Representation,” 2024.

3. I cannot see essential advantages of the proposed generation paradigm, because it still falls into the scope of multi-view image diffusion. Note that in the original design of Splatter Image, it can encode the whole 3D scene within a single image by modeling Gaussian position as a combination of depth and offset. Yet in this paper the Splatter Image still uses (x, y, z) coordinates and requires multi-view representations. Thus, the joint modeling of multi-view consistency is still a major challenge. Besides, the pre-trained diffusion priors are somewhat damaged, because the VAE component (the decoder) need to be retrained.

4. The ground-truth Splatter Image representations are generated by LGM, which fundamentally limiting the potential of the proposed method.
Yes
No ethics review needed.
Please refer to the weaknesses above.
1. Due to the scarcity of 3D datasets, this paper presents a method to unleash the power of pretrained 2D diffusion models for direct 3D generation, with the potential to further enhance the performance of 3D generation.
2. The paper is generally well-written and easy to follow. The authors clearly explain the motivation behind their work, and the key components of their proposed method. The figures are helpful in illustrating the overall fine-tune VAE+LDM pipeline.
3. The experimental results show better performance than baselines.
4. The paper includes relevant ablations validating the individual design choices.
This paper introduces a new approach called ZERO-1-TO-G for 3D generation using Gaussian splats through 2D diffusion models. The key contribution of this paper is to represent 3D objects as multi-view, pixel-aligned Gaussian images and to fine-tune the VAE decoder so that the 3D objects can be generated directly. Experiments on GSO demonstrate promising results, with the proposed method outperforming several baselines in image-conditioned 3D generation tasks.
1. My main concern is that, despite the use of cross-view and cross-attribute attention layers to enhance 3D consistency, the generated results still exhibit inconsistencies across multiple views. How can we address these inconsistencies in the multi-view Gaussian splat results?
2. Methods such as 3DShape2Vec and 3DILG conduct experiments on shape autoencoding, which serves as the upper bound for the generated 3D objects. I believe that this experiment is also critical for ZERO-1-TO-G.
3. Missing comparisons experiment of generated 3d models for text-to-3d.
4. Missing reference: 
[1] Cat3d: Create anything in 3d with multi-view diffusion models. NIPS 2024
[2] GPLD3D: Latent Diffusion of 3D Shape Generative Models by Enforcing Geometric and Physical Priors. CVPR 2024
Yes
No ethics review needed.
See Weaknesses.
* The paper proposes a novel framework for 3D generation by finetuning a pretrained 2D diffusion model for directly 3D generation, which is novel to me.
* The experiments are comprehensive and well-executed.
* The paper is well-written and easy to understand for people familiar with 3D generation tasks.
This paper presents a novel method for 3d generation by finetuning a pretrained 2D diffusion model to generate 3D gaussian splatter images directly according to a single-view input image. The method integrates the generative priors from the pretrained diffusion model into the task of direct 3D generation and achieve good results.
* The paper adopts LGM to build to overall dataset. This raises the concern of the overall quality of the training dataset. In line 216-217, the author declares that fitting-based reconstruction methods lead to excessively high-frequency signals which is hard for pretrained VAE to decode. However, in the paper, the author actually finetuned the decoder part of VAE with LGM outputs, which is confused. Why we could not finetune a VAE with the  fitting-based reconstruction data? The author should clarify this.
* The discuss of Limitation is not included in the paper.
* Overall, representing and generating Gaussian as images have been proposed by Splatter Image(CVPR2024), so the technological contribution of this paper is relatively limited. But I like the idea of finetuning a diffusion model to enjoy the generative priors, so I vote to a borderline accept.

[1] Splatter Image: Ultra-Fast Single-View 3D Reconstruction (https://arxiv.org/abs/2312.13150
Yes
N/A
No ethics review needed.
See weakness
* Decomposing the 3D Gaussian splat generation problem into multi-view and multi-attribute image generation is interesting.
* The paper is structured well, making it easy to understand and follow.
This paper introduces Zero-1-to-G, a novel 3D generative framework that generates 3D Gaussian splats from a single input image. The framework decomposes the generation process into five distinct attribute generations for fixed camera viewpoints, allowing a fine-tuned 2D diffusion model and a VAE decoder to generate multi-view, multi-attribute images.
**Weakness 1**

The primary concern is the novelty of the paper. LGM [1] is designed to quickly generate Gaussian splats from various multi-view images, with the advantage of compatibility with different multi-view diffusion models. So, I think Zero-1-to-G can be regarded as a combination of an existing multi-view diffusion model and LGM, extending the cross-domain attention mechanism of Wonder3D [2] to five attributes.

**Weakness 2**

Experimental results are not sufficient to prove the effectiveness of Zero-1-to-G. Many One-Image-to-3D [2, 3, 4, 5] models report 3D reconstruction results (e.g. Chamfer Distance) on the GSO dataset. Since Zero-1-to-G takes 8.7 seconds to generate a 3D Gaussian splat, it would be better to show the performance with fewer sampling steps and a similar inference time to LGM.

[1] Tang et al., LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation, ECCV 2024.

[2] Long et al., Wonder3D: Single Image to 3D using Cross-Domain Diffusion, CVPR 2024.

[3] Woo et al., HarmonyView: Harmonizing Consistency and Diversity in One-Image-to-3D, CVPR 2024.

[4] Liu et al., SyncDreamer: Generating Multiview-consistent Images from a Single-view Image, ICLR 2024.

[5] Xu et al., DMV3D: DENOISING MULTI-VIEW DIFFUSION USING 3D LARGE RECONSTRUCTION MODEL, ICLR 2024.
We present Zero-1-to-G, a novel native 3D diffusion model for Gaussian splats using multiview splatter images.
@inproceedings{
anonymous2024zerotog,
title={{ZERO}-1-to-G: Taming Pretrained 2D Diffusion Models for Direct 3D Generation},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=nmc9ujrZ5R},
note={under review}
}
Recent advances in 2D image generation have achieved remarkable quality, largely driven by the capacity of diffusion models and the availability of large-scale datasets. However, direct 3D generation is still constrained by the scarcity and lower fidelity of 3D datasets. In this paper, we introduce Zero-1-to-G, a novel approach that addresses this problem by enabling direct 3D generation on Gaussian splats through 2D diffusion models. Our key insight is that Gaussian splats, a 3D representation, can be decomposed into multi-view images encoding different attributes. This reframes the challenging task of direct 3D generation within a 2D diffusion framework, allowing us to leverage the rich priors of pretrained 2D diffusion models. To incorporate 3D awareness, we introduce cross-view and cross-attribute attention layers, which capture complex correlations and enforce 3D consistency across generated splats. This makes Zero-1-to-G the first direct 3D generative model to effectively utilize 2D pretrained diffusion priors, enabling efficient training and improved generalization to unseen objects. Extensive experiments on both synthetic and in-the-wild datasets demonstrate superior performance in 3D object generation, offering a new approach to high-quality 3D generation.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
native 3D diffusion 3D generative model gaussian splats
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/fbac48850868cb9b65091b0d1679ef3e1ebb3778.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
ZERO-1-to-G: Taming Pretrained 2D Diffusion Models for Direct 3D Generation
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission