Layout-your-3D: Controllable and Precise 3D Generation with 2D Blueprint
3D generation;gaussian splatting;Text-to-3D;compositional asset generation
Yes
No ethics review needed.
Within the collision loss, is there any way to take into account the shape (covariance) of each gaussian in 3D space? My understanding is that it's only using the positions of the the Gaussians right now. Have you observed this to be a limitation?
The papers shows a potential way to use many existing models together to generate compositional 3D scenes. The results show that the method produces reasonable visual reconstructions and geometries. The method seems well motivated, and the paper is decently written.
- They enforce structural control of a 3D scene using a 2D layout. Their optimization process avoids collisions.
- Their method works by first generating a 2D image based on text description, then use SAM to segment each isntnace in the reference iamge, with in-painting done on occluded parts.
- They then use a large mulit-view gausian model to generate coarse 3D instances of objects within the scene. 
- Each 3D instance is then arranged into 3D space by lifting the bounding boxes into 3D, done by optimizing the rotation according to similarity in the DINO space of the rendered images of the rotated instances.
- final two stages refine the layout according to collisions and refines the instance representations.
My concern is that this work lacks fundamental novelty, in that prior works like Compositional 3D Scene Generation using Locally Conditioned Diffusion (https://arxiv.org/abs/2303.12218), Towards Text-guided 3D Scene Composition (https://arxiv.org/pdf/2312.08885) and CG3D: Compositional Generation for Text-to-3D via Gaussian Splatting (https://arxiv.org/pdf/2311.17907) all present various ways to create 3D scenes using implicit 3D represntations in a compositional way.

There does not seem to be any comparisons to these works, or mention of them in the related works (though CG3D was mentioned in the collisions section, but not compared with in the experiments section). A comparison to contextualize this work seems very important, but lacking.
Yes
No ethics review needed.
1. For clarification, when inputing the text prompts, what's the prompt for LLM to generate 2D layouts? Also, do you mark the object in bracket like text prompts in figure 5 or LLM is supposed to decide which objects should be created?

2. The proposed method uses many off-the-shelf algorithm and components like LLM generating 2D layouts, reference image generation, pose alignment. I am curious about the successful ratio of the whole process, and some potential failure cases.
1. This paper proposes an interesting pipeline by combining several (off-the-shelf) components in a novel way. Getting 2D reference images from layout and trying to reconstructing 3D models from them leads a interesting 3D reconstruction task, and this paper provides a reasonable solution for it, by utilizing the inpainting, DINO feature distance and off-the-shelf image generation model.
2. This paper involves several intesting losses like collision-tolerant loss, feature-level loss, and SDS with adjusted timesteps. 
3. Results are good. and the paper is easy to read and follow.
This paper tries to address the problem of lay-out controlled 3D generation task. 
Interestingly, it relies on 2D layout instead of 3D layouts, and the 2D layout could be generated by LLM given a text prompt. 
With this 2D Layout, it first generate a reference image with an off-the-shelf 2D-layout image generation model, and then try to reconstruct the 3D model and layout from the reference image. 
This generated 3D model is coarse and may not very accurate, and this paper refines the generated model with SDS and feature loss, as well as collision-tolerant loss.
This paper proposes a very interesting pipeline using many off-the-shelf components, and it achieves good results
1. For baselines, it would be more convincing to seriously compare against models with 3D layout control like ComboVerse or [Compositional 3D Scene Generation using Locally Conditioned Diffusion]. The current paper only shows a single scene comparison against ComboVerse in Figure 7, which is insufficient to demonstrate the advantages of using 2D layout and the proposed method. More rigorous evaluation, such as those shown in Tables 1 and 2, along with a user study, would be needed for such comparisons.

2. After reconstructing individual 3D models from 2D images, it is critical to align the pose (6D pose) of the reconstructed 3D model with the image appearance. The proposed method achieves this by randomly sampling many views of the 3D model and comparing them against the reference image by computing the DINO feature distance. However, I am curious about the accuracy of this method, as the rendered image could differ significantly from the reference image even with perfect poses if the distance (object scale) is not accurate. It would be more convincing to show results demonstrating the accuracy of this method.

3. Regarding the reconstruction of objects from reference images, how is the mask/segmentation obtained for each object? Is this process automated, or does it require manual intervention? This question also appears in the following inpainting and pose alignment sections. Do we need to manually specify which object to inpaint or segment during this process or not?
Yes
No ethics review needed.
1. why do the bird, bear, and Bagle look different in each example?
2. what happens if the generated 2D images don't contain all the specified objects? it does happen a lot with the generation of complex scenes. like what is shown in [2]
generating complex scenes is a difficult task today.
this work tries to solve this task, into stages to deal with each part separately.
the work trying to generate a 3D complex scene based on a textual prompt. they start by generating a 2D image of the scene using a 2D text-to-image generator, they extract 2D boxes for each instance. 
Next, they create each object in a 3D instance and position them according to the boxes.
the final step is SDS which fine-tune the positions for natural placement, specifically in a collision-free manner.

[1] Yang, Yue, et al. "Holodeck: Language guided generation of 3d embodied ai environments." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024
[2] Rahamim, Ohad, et al. "Lay-A-Scene: Personalized 3D Object Arrangement Using Text-to-Image Priors." arXiv preprint arXiv:2406.00687 (2024).
1. no novelty in the approach, you concatenated known strategies while trying to solve this problem.
2. can it be solved using previous methods like [1]
3. Besides the bear and the book, I can't see any interaction between objects. can you propose how to create interactions? (like the bird drinking the beer)
Yes
No ethics review needed.
- My main concerns are discussed in the Weaknesses section. I can see from the paper that the authors have put a lot of effort into creating a valuable contribution, and despite my doubts about the contribution, experimental results, and design of the method, I still rate the paper as borderline reject (5). I believe the improvements needed cannot realistically be achieved during the rebuttal stage, and this may be due to time constraints in preparing the submission. I hope the authors can improve the method’s effectiveness and rewrite the contribution section to better highlight their approach, so that the paper can be successfully accepted at top conferences like **CVPR, ICML, or ICCV**.

- One final note: while I don't mean to discourage the authors, I have limited expectations for changes in the rebuttal, as the scope of improvement I am suggesting is quite large. Therefore, **it might not be necessary to spend too much time adding experiments during rebuttal for me**. I wish the authors all the best with their research.
- The paper is rich in content, and the proposed method successfully accomplishes the task set by its motivation.
- The writing is clear, making it easy to understand the authors' research methods.
This paper uses a 2D layout as guidance and proposes a Collision-aware Layout Refinement to optimize 3D layouts, effectively achieving compositional 3D generation with plausible object interactions.
- Over-reliance on existing methods: The engineering aspect of the method is too pronounced. Aside from the newly introduced loss functions, much of the technique is a stack of existing methods, including SAM, ControlNet-tile, LGM, GeoWizard, DINO, etc. The use of LLM to assist in generating the 2D layout is also an existing strategy. Such extensive reliance on pre-existing methods can overshadow the key contribution of the paper. Additionally, I would like to ask: since LLMs are already in use, why not have the LLM or MLLM directly predict the 3D layout and then apply your Collision-aware Layout Refinement for 3D layout refinement? This would simplify the initial stage, reduce reliance on existing methods, and highlight the contribution of your technique, making the overall framework more elegant and concise.

- Limited contribution: The task goal overlaps significantly with many existing compositional 3D scene generation methods (GALA3D, graphdreamer, dreamscene， progress3D...), making the problem being addressed less novel. Furthermore, as mentioned earlier, many critical components are stacked from existing methods. Although the Feature-level Reference Loss and Tolerant Collision Loss are novel, their impact seems less substantial compared to the other parts, as seen in the visual results in Figure 8. As a suggestion for improvement, it might be better to focus on complex single-image-to-3D reconstruction instead of compositional 2D layout + text-to-3D generation. You could use multimodal LLMs to decompose objects from a complex, unclear image for reconstruction and then refine and combine them into a 3D scene. This shift could substantially enhance the novelty and contribution of the paper.

- Insufficient comparison in experiments: In the supplementary materials, methods like GALA3D and DreamScene are mentioned. Why aren’t there comparisons with compositional 3D scene generation methods in the main text, such as GALA3D, GraphDreamer, DreamScene, Progressive3D, or more recent works like The Scene Language? Even if comparing with the very recent papers isn't necessary, it would be more reasonable to compare your work with three or so methods that are highly relevant to your contribution, such as GALA3D, GraphDreamer, and DreamScene. This could also help improve the previous point about novelty. Additionally, based on my own experience, DreamFusion should be faster than LucidDreamer—on my computational resources, the timing is approximately 30 minutes vs. 50 minutes, differing from the 1 hour vs. 1 hour stated in your paper. Although a minor issue, it raises some doubts about the accuracy of the reported experimental results.
Given a text prompt describing multiple objects and their spatial relationships, our method generates a 3D scene depicting these objects naturally interacting with one another.
@inproceedings{
anonymous2024layoutyourd,
title={Layout-your-3D: Controllable and Precise 3D Generation with 2D Blueprint},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=myolhJPuRI},
note={under review}
}
We present Layout-Your-3D, a framework that allows controllable and compositional 3D generation from text prompts. Existing text-to-3D methods often struggle to generate assets with plausible object interactions or require tedious optimization processes. To address these challenges, our approach leverages 2D layouts as a blueprint to facilitate precise and plausible control over 3D generation. Starting with a 2D layout provided by a user or generated from a text description, we first create a coarse 3D scene using a carefully designed initialization process based on efficient reconstruction models. To enforce coherent global 3D layouts and enhance the quality of instance appearances, we propose a collision-aware layout optimization process followed by instance-wise refinement. Experimental results demonstrate that Layout-Your-3D yields more reasonable and visually appealing compositional 3D assets while significantly reducing the time required for each prompt. Additionally, Layout-Your-3D can be easily applicable to downstream tasks, such as 3D editing and object insertion.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
3D generation gaussian splatting Text-to-3D compositional asset generation
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/9dcf442ed9a0dbed38b98e762e52ef2914888643.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/1ba4136f6698c133828c6b2bc3d69527787d14cd.zip
Layout-your-3D: Controllable and Precise 3D Generation with 2D Blueprint
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission