Tailor3D: Customized 3D Assets Editing and Generation with Dual-Side Images
3D Object Editing;3D Generative Models
Yes
No ethics review needed.
* Following the point mentioned in the weakness seciton, I wonder whether it is possible to extend or generalize the proposed method to the future 3D generative model? Since I think if not, then for real-world scenario, the proposed method is not useful at all because of limited 3D content quality bounded by the backbone model.
*
* The input on both front-view and back-view seem reasonable and achieve good quality results.
* The evaluation seems thorough.
This paper presents a method that allows users to customize 3D assets using 2D edits through a dual-side triplane-NeRF generation model that takes both front-view and back-view as input. And they perform thorough studies to evaluate the proposed method.
* The method is a combination of existing methods and trained on a new data format that involves both front and back view. For me, it is unclear what is the major technical contribution.
* The quality of the generated 3D results are poor... I think this is due to the backend (LRM), but I still think it is worth pointing out that so many people nowaday relying on these poor 3D generative model for different purpose, but the reality might be that the quality of the 3D content generated by these models are just not good enough for real-world applications.
Yes
No ethics review needed.
I have some questions listed in the weakness part. I would be very happy to change my rating given the answers of these questions.
1.This paper introducing a very interesting sub-problem: given a pretrained LRM, how could we merge the the output triplane of dual-side image into a single triplane. This paper proposes a sound method to do so and shows the effectiveness in the experiments.
2. The ablation study in this paper is quite comprehensive, including the way of merging two triplanes, the rank of LORA, extrinsic parameters, etc.
Also, this paper provides the failure cases and analysis, which is very useful for understanding the model.
3. This paper is easy to read and follow.
This paper proposes Tailor3D, a method for fast 3D generation and editing. 
The core component of Tailor3D is a dual-side image LRM, which takes a dual-side image pair (front and back) as input, and output a 3D representation of the object (in Triplane).
Compared to original LRM, this dual-side LRM takes two images, instead of a single image or four images.
The motivation of using dual-side image is that, using two images, especially front and back views can provide convinience for users to edit while preventing critical inconsistencies from more views (like 4 views). 
This paper show extensive experiments about dual-side image LRM, including generation, editing, and very dense ablations.
1. My biggest concern about this paper is that many considerations lack clear motivation, which makes me wonder whether some problems addressed are real or artificial.

(1) This paper emphasizes reducing the memory usage of LRM and therefore proposes using LoRA for a pretrained LRM. However, I don't see a clear reason why memory usage would be problematic here. Given that there are even four-view versions of LRM [DMV3D: DENOISING MULTI-VIEW DIFFUSION USING 3D LARGE RECONSTRUCTION MODEL] and this paper uses an A100 GPU, I don't understand why memory usage is a critical concern and whether LoRA is a necessary component for the model.

(2) Similarly, this paper proposes a novel method to merge two triplanes using cross-attention. I like this solution. However, I don't see a clear justification for this approach. In particular, why not simply take the features from both images and fine-tune the LRM model to generate a single triplane? This pipeline seems more natural and easier to implement. I may be missing some important considerations here.

(3) The dual-side image setting also seems somewhat unusual. While I agree that using two images can help eliminate inconsistencies across multiple views, it also constrains general editing capabilities. For example, if users want to edit the side of an object (between the front and back views), the current setting may be inconvenient. This raises the question: is the current setting really necessary? Could we instead use four views, edit in one of them, and fine-tune LRM to propagate these changes into triplanes?

It would be greatly appreciated if the authors could address these questions in their rebuttal. More importantly, the paper would benefit from clearer motivation and more detailed elaboration of these considerations to help readers better understand both the problems and proposed solutions.

2. Table 1 is confusing to me. What's the setting of single-image generation here? How could the proposed method achieve this task as the trained LRM takes dual-side images as input? If just providing single image, why this model outperforms the others so much as it should be similar to the original LRM with single image input?

3. I understand that training LRM may take a long time and many resources. But I still think it might be worthwhile to do some higher-level ablations to support some design considerations, mentioned in the first point of weakness. For example, could we train a LRM taking the image embedding from two images and directly outputs the triplane? Would that be better or worse compared to the current one? I think this paper does a great job on ablating the components in the current design, but some higher-level ablations would be really needed to support some decisions.
Yes
None
No ethics review needed.
1. Compared to editing in a multi-view diffusion model, what are the advantages of editing in a single view? From my perspective, editing in a single view invariably incurs inconsistencies, though only front and back views are used, there is still an overlapped area for some objects.

2. Compared to training an LRM with only two views as input, what are the advantages of the proposed LoRA Triplane Transformer, except for memory cost? For example, how about the performance comparison? Moreover, since the LoRA Triplane Transformer is based on a pretrained LRM, how does the performance of directly training a Dual-side LRM from scratch compare?
1. The proposed Dual-LRM and the editing of front and back views are technically sound. Editing these views without significant overlap effectively avoids inconsistent results. The proposed Dual-LRM can transform the two views into a 3D mesh.
2. The manuscript presents extensive editing results and an ablation study to validate each component within the framework.
3. The proposed LoRA Triplane Transformer and Fuse Double Side Feature effectively reduce the finetuning cost based on a pretrained LRM.
This manuscript introduces a framework for generating customized 3D assets, which integrates 2D image editing methods with a Dual-side LRM. Initially, the front view is edited using 2D editing techniques. Subsequently, the back view is generated by a multi-view diffusion model and can also be edited using the same 2D method. Both views are then processed by the Lora Triplane Transformer to create the 3D object. The effectiveness of this method has been validated through extensive applications.
1. Grammar typo in Line 71.
2. Use \citep for citation.
3. The motivation in the contribution section is not clear. Dual-LRM is proposed to avoid inconsistency in multi-view image editing results. Why is there still inconsistency between the front and back views?
4. Too few comparison methods. InstantMesh[1] and CRM[2] also leverage multi-view images as input for 3D generation.
5. The technical novelty is a minor concern. The 2D image editing and multi-view diffusion model are directly used. The dual-sided LRM can also be directly replaced with InstantMesh. I recommend a comparison with InstantMesh by inputting edited two views into InstantMesh and Dual-side LRM.

[1]Xu J, Cheng W, Gao Y, et al. Instantmesh: Efficient 3d mesh generation from a single image with sparse-view large reconstruction models[J]. arXiv preprint arXiv:2404.07191, 2024.

[2] Wang Z, Wang Y, Chen Y, et al. Crm: Single image to 3d textured mesh with convolutional reconstruction model[J]. arXiv preprint arXiv:2403.05034, 2024.
Yes
No ethics review needed.
My questions are listed in the weakness section
This is a nice system paper with quite some strengths
- The overall presentation of the approach is accompanied by good visualizations and diagrams. I find it useful understanding the whole paper
- The technique is interesting and it nicely integrated several technologies such as LRM, LoRA, and efficient attention mechanisms to edit and combine feature representations of front and back views. 
- Overall, the result is competitive comparing to other approaches, it's nice to show user studies rather than automatically computed benchmark numbers in this paper
The paper introduced a practical system approach for 3D editing with the help of multi-view generations. The approach Tailor3D by generating the back view, then stitch the front and back view features for fine-tuning the reconstruction results. When testing on several datasets and benchmarks, the user studies showed the effectiveness of the approach, and it was quite competitive.
I think this is a nice system paper with good quality results. There are strong merits of this paper but I have a few questions
- I think the approach is heavily bounded by the quality of back-view generation algorithms. I wonder if authors could comment on whether there are empirical evidences or ablation studies on this topic. E.g: if the back-view generation algorithm is not performing well, would the front & back view fusion approach recover the issues?
- The approach is limited to using 2 views (front and back), would using more views increase the complexity of the formulations in this paper?
- In eq. 4, could authors help elaborating on the theory of the equation, or point to sections of the LRM paper that explains this equation? I think it might sound intuitive but I wonder if this is mathematically correct.
Tailor3D swiftly generates customized 3D assets from edited dual-side images and offers efficient editing solutions for diverse 3D tasks.
@inproceedings{
anonymous2024tailord,
title={Tailor3D: Customized 3D Assets Editing and Generation with Dual-Side Images},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=SJqoP0rp8w},
note={under review}
}
Recent advances in 3D AIGC have shown promise in directly creating 3D objects from text and images, offering significant cost savings in animation and product design. However, detailed edit and customization of 3D assets remains a long-standing challenge. Specifically, 3D Generation methods lack the ability to follow finely detailed instructions as precisely as their 2D image creation counterparts. Imagine you can get a toy through 3D AIGC but with undesired accessories and dressing. To tackle this challenge, we propose a novel pipeline called Tailor3D, which swiftly creates customized 3D assets from editable dual-side images. We aim to emulate a tailor's ability to locally change objects or perform overall style transfer. Unlike creating 3D assets from multiple views, using dual-side images eliminates conflicts on overlapping areas that occur when editing individual views. Specifically, it begins by editing the front view, then generates the back view of the object through multi-view diffusion. Afterward, it proceeds to edit the back views. Finally, a Dual-sided LRM is proposed to seamlessly stitch together the front and back 3D features, akin to a tailor sewing together the front and back of a garment. The Dual-sided LRM rectifies imperfect consistencies between the front and back views, enhancing editing capabilities and reducing memory burdens while seamlessly integrating them into a unified 3D representation with the LoRA Triplane Transformer. Experimental results demonstrate Tailor3D's effectiveness across various 3D generation and editing tasks, including 3D generative fill and style transfer. It provides a user-friendly, efficient solution for editing 3D assets, with each editing step taking only seconds to complete.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
3D Object Editing 3D Generative Models
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/f11d28d72bdd325877d1ec502a5373979497ccb3.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/f15834a2ca80a32379fd47b442df199c11588cf0.zip
Tailor3D: Customized 3D Assets Editing and Generation with Dual-Side Images
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission