Geometry Image Diffusion: Fast and Data-Efficient Text-to-3D with Image-Based Surface Representation
Text-to-3D; Geometry Image; Non-watertight mesh; Data efficiency
Yes
No ethics review needed.
Please see the first two questions in the weakness: the chart quality and diversity; and the artifacts on the boundary.
- The reviewer loves the elegant idea very much. I personally believe this is the correct way in a long-term to generate usable (industry level) and high-quality assets for objects. Because most generative model for objects now depends on large synthetic dataset, with known geometry.
- The system is simple and effective.
- Also, the semantically meaningful parameterization has much more space in the future for editing and manipulation.
This paper studies how to generate high quality mesh of objects with GIM.

The 3D objects are parameterized into 2D chart geometry image. This representation is compact where the prior of 2D SD model can be exploited and maintains good geometry and semantic properties. The generation happens vis modifying standard SD with cross attention between different attributes of the chart.
- My main concern is maybe the diffusion overfit the proposed UV chart distribution. But we know that there is uncertainty of how an object can be parameterized into a chart. Is there any measurement of how good the chart is in the proposed preprocessing pipeline? e.g. it maintains the surface area or local differential properties? Can the proposed method generate different chart pattern for the exact same object? And how diverse is the chart?
- There are some small artifacts in the boundary of the mesh, which may due to the boundary of the chart. I believe this can be fixed in future works, but an analysis of this artifacts may provide more intuition of the future work.
- Maybe one or two applications like relighting .etc will future demonstrate the strength of generating object via GIM.
Yes
No ethics review needed.
1. How does the variability in multi-chart geometry image layouts impact GIMDiffusion’s training stability and scalability?
2. The quality of shape edges appears to be suboptimal in the generated results. Is this a known limitation of the geometry image representation, and are there potential solutions to address this?
3. The authors freeze the albedo generation branch based on the pre-trained SD model, but SD’s performance in UV space is often limited. Could the authors provide experimental evidence to show that this decision does not compromise the quality of generated RGB textures?
1. Geometry images provide a low-complexity 3D representation, allowing GIMDiffusion to fully leverage pre-trained 2D diffusion models for 3D generation.
2. The model preserves the weights of the pre-trained image diffusion model, making it compatible with plugins like IP-Adapter for style transfer and customization.
The paper introduces Geometry Image Diffusion (GIMDiffusion), a novel Text-to-3D generation model that uses geometry images to represent 3D shapes in a 2D format. This approach leverages the priors of pre-trained image diffusion models, allowing efficient 3D object generation without complex 3D-aware architectures. By retaining pre-trained weights, GIMDiffusion is compatible with tools like IP-Adapter, enabling flexible style control and broad applicability. The use of multi-chart geometry images further enhances the model’s ability to represent complex shapes, offering a scalable and versatile solution for text-driven 3D generation.
1. My primary concern with multi-chart geometry images is the variable layout configurations that a single 3D shape can produce. That means a single 3D shape may correspond to many different geometry images, potentially complicating the training process and impacting consistency across different shapes. Is any technique used to standardize the conversion from mesh to geometry image.
2. Generated results in the paper often show lower quality around shape edges, possibly due to the geometry image representation. This limitation affects the clarity and sharpness of complex geometries.
3. The model freezes the pre-trained Stable Diffusion (SD) model for generating albedo textures. However, SD has known limitations in UV space, and freezing this branch could compromise RGB quality. Providing experiments to validate this decision would help substantiate the model’s robustness.
Yes
No ethics review needed.
1. Could you include comparisons with more recent Text-to-3D approaches, such as MVDream, CRM, DreamGaussian, and other fast, high-quality methods based on Gaussian Splatting? Including comparisons to more recent approaches would offer readers a clearer understanding of where GIMDiffusion stands relative to current advancements.

2. How does GIMDiffusion perform with diverse object types, especially those with intricate internal structures or shapes? Could you provide more detail on potential methods to mitigate visible seams or artifacts between geometry images? It would be helpful if you could discuss or experiment with approaches for reducing these seams, such as stitching algorithms or blending techniques that could improve texture continuity, especially for high-fidelity applications.

3. The visual quality of this method is not obviously better than some other recent methods, which raises concerns about its competitive edge. While GIMDiffusion offers efficiency and compatibility with 2D architectures, it would benefit from additional improvements or evaluations to enhance the clarity, realism, and continuity of generated textures and geometry.
1. Innovative 2D-Based Approach for 3D Generation: By using geometry images, GIMDiffusion avoids the need for specialized 3D architectures, simplifying the overall system and reducing the computational resources required.
2. The paper introduces a Collaborative Control mechanism, which allows two parallel diffusion models to communicate effectively. This cross-model communication lets the geometry generation model inform and guide the 2D texture model, a unique architectural contribution that enhances model efficiency and adaptability.
3. Rapid and Resource-Efficient Processing: The generation time is highly competitive, making this model a viable choice for real-time applications and scalable deployment.
1. GIMDiffusion uses geometry images to represent the surface of 3D shapes in a 2D format. This method, unlike traditional 3D meshes, can utilize the extensive 2D image generation capabilities of diffusion models like Stable Diffusion. This approach results in faster generation times, lower data requirements, and the elimination of the need for complex 3D-specific model architectures, which are often cumbersome and data-hungry.

2. The model introduces a Collaborative Control mechanism to integrate 2D Text-to-Image models with geometry images. Through this mechanism, the geometry and texture generation processes are interconnected, allowing each to influence the other. This setup ensures that the textures generated align well with the 3D shape, facilitating realistic and coherent representations even when there is limited 3D data available.

3. Efficiency and Performance: GIMDiffusion achieves faster-than-average generation times, producing complex 3D assets in approximately 10 seconds. Moreover, the model avoids intermediate 3D processing stages like iso-surface extraction, further streamlining the pipeline.
1. In figure 9, the paper observes artifacts, such as visible seams between geometry images, which could compromise visual quality, especially in high-precision applications requiring seamless textures and continuous surfaces.

2. While the paper highlights the efficiency and practicality of geometry images, it doesn’t fully address potential limitations of this 2D representation for complex or high-genus shapes. Multi-chart geometry images mitigate some challenges, but further analysis is needed on cases where this approach may fail, such as in scenarios with high geometric distortion or disconnected regions.

3. Limited Comparison with Recent Methods: The paper's evaluation lacks comprehensive comparisons with the latest advancements, instead focusing on older benchmarks like DreamFusion, ProlificDreamer, and Shap-E. Expanding comparisons to include recent approaches, such as MVDream, CRM, DreamGaussian, and Text-to-3D using Gaussian Splatting, would clarify where GIMDiffusion specifically excels or encounters limitations.
Yes
No ethics review needed.
I noticed that the albedo branch used the pretrained VAE, buth GIM is trained from scratch. Have the authors considered also using the pretrained model for GIM? What is the reason of not using the pretrained model? Is there any comparison for that?
- The idea of using pretrained diffusion models for native 3D generation is interesting and novel
- The proposed method is straightforward and easy to understand. The proposed components are also well validated in the ablation study.
- The generated 3D meshes have semantic information and can be decomposed into several parts, make it potential for downstream editing tasks. It also directly generates UV maps and is directly relightable.
This paper presents a method to generate geometry images with diffusion models, which represents meshes with texture maps with images. Specifically, the authors finetuned stablediffusion to generate the albeda images and use collaborative control to generate the geometry images, which can then be unfolded to 3D meshes. The authors evaluated their method on T^3 bench and shows that it can generally well beyond training data and also have better geometry.
- Although the generated meshes are quite sharp and have details, there are obvious discontinuous artifacts, which can be observed in the video. The reason might be that during reconstructing the ground truth geometry images, these parts are already disconnected. Also, in Fig. 7, the boundaries of the mesh also have noticeable artifacts. What is the reason behind these artifacts? Are these artifacts solvable by improving the current pipeline? Is it possible to apply some post-processing techniques to deal with that?
- The details of the evaluation is not that clear: how many prompts are used during evaluation and what are they? The authors may provide the number of prompts and some examples of that.
- I would like to see more comparison with two-stage mesh methods, e.g. first multi-view and then reconstruct mesh, especially CRM, which works faster and have better quality than LGM to my knowledge. It can also produces UV maps. Besides, the authors should also show more examples of comparison in the paper. It would be great if the authors could provide the same comparisons similar to what have been shown in Table 1 for CR.
We propose a fast and data-efficient Text-to-3D model using geometry images as the surface representation.
@inproceedings{
anonymous2024geometry,
title={Geometry Image Diffusion: Fast and Data-Efficient Text-to-3D with Image-Based Surface Representation},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=Glm7Kj47nN},
note={under review}
}
Generating high-quality 3D objects from textual descriptions remains a challenging problem due to high computational costs, the scarcity of 3D data, and the complexity of 3D representations. We introduce Geometry Image Diffusion (GIMDiffusion), a novel Text-to-3D model that utilizes geometry images to efficiently represent 3D shapes using 2D images, thereby avoiding the need for complex 3D-aware architectures. By integrating a Collaborative Control mechanism, we exploit the rich 2D priors of existing Text-to-Image models, such as Stable Diffusion, to achieve strong generalization despite limited 3D training data. This allows us to use only high-quality training data while retaining compatibility with guidance techniques such as IPAdapter. GIMDiffusion enables the generation of 3D assets at speeds comparable to current Text-to-Image models, without being restricted to manifold meshes during either training or inference. We simultaneously generate a UV unwrapping for the objects, consisting of semantically meaningful parts as well as internal structures, enhancing both usability and versatility.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Text-to-3D; Geometry Image; Non-watertight mesh; Data efficiency
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/e8df6b23fbf46d3ccf9c5d7c6a40293f8b03445f.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/f3967294de66fc93f8381735ecb3a4a457ac58b8.zip
Geometry Image Diffusion: Fast and Data-Efficient Text-to-3D with Image-Based Surface Representation
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission