InfiniteMesh: View Interpolation using Multi-view Diffusion for 3D Mesh Reconstruction
View Interpolation;Multi-view Diffusion;3D Mesh Reconstruction
Yes
No ethics review needed.
1. How many interpolated views are used in total to optimize the mesh reconstruction? From what I understand, Wonder3D initially produces 4 views from a single image, and then the IVI module generates n = 2 interpolated views each on left and right for these 4 main views - so in total, 20 views are used for reconstruction (16 interpolated and 4 from Wonder3D) Is that correct?
1. The IVI module is a novel contribution that improves multi-view consistency by generating interpolated views between 4 main views generated by Wonder3D. This approach minimizes inconsistencies across views, addressing a common challenge in 3D mesh generation. 

2. The combination of multi-view interpolation and a tri-plane-based reconstruction model results in superior mesh quality, with enhanced detail in both geometry and texture. The framework effectively captures intricate features and realistic textures, outperforming current state-of-the-art models both qualitatively and quantitatively.
The paper introduces InfiniteMesh, a framework for generating high-quality 3D meshes from a single image by combining multi-view diffusion-based view interpolation and mesh reconstruction techniques. The main task is to create consistent, high-fidelity 3D representations from limited initial views, addressing common issues like inconsistency and low detail in existing methods. Key contributions include the Infinite View Interpolation (IVI) module, which enhances multi-view consistency by generating interpolated views between initial main views, and a tri-plane-based reconstruction model that processes these views to produce accurate 3D meshes. Experimental results show that InfiniteMesh outperforms state-of-the-art approaches in both geometric and textural quality, especially on the Google Scanned Objects dataset.
1. Inference Time missing: Infinite Mesh reports stronger quantitative performance and slightly better geometry and texture synthesis than several baselines. However, I am concerned about the extra computation time incurred by the IVI module for synthesizing sufficient novel views for multiview consistency. Instant Mesh constructs a mesh in 10 seconds, I doubt whether this method can get close to such speeds given the IVI overhead. For a fair comparison, the authors should report and compare average mesh creation times with the baselines considered in the paper.
2. The ablation with the camera trajectories for the IVI module does not seem too convincing, both qualitatively and quantitatively. The NVS results for the 2 hand-picked novel views are mostly similar, and the meshes have similar geometry details.
3. More details on mesh reconstruction: The authors should ideally provide more details and the motivation behind specific loss functions for mesh reconstruction in Equation 7, either in the main paper or the supplementary. This would save the reader (unfamiliar with all related literature, like the RODIN paper) from spending too much time reading other papers to understand the method correctly.
4. No 360 reconstruction videos: The authors should ideally share 360 reconstruction videos in the supplementary so that the multiview consistency introduced by the LRM and IVI modules can be better evaluated. Renderings of a few hand-picked novel views do not do much to convince the reader. The IVI module is introduced as a more compute-friendly alternative to video diffusion models for ensuring multiview consistency. However, current qualitative results do not support that entirely.
Yes
No ethics review needed.
- In equation 3, where $n$ is defined?
- As the number of input images increases for the LRM model, how does the time required for training and inference compare to that of the sparse-view version of the LRM model?
- Also, as the number of output images increase for the multi-view generation model,  how does the time required for training and inference compare to that of the sparse-view version of multi-view generation model?
- This paper is easy to read with clean writing.
- The problem definition is clear.
- The paper compares the recent state-of-the-art image to 3D models, which is good.
This paper introduces a pipeline for reconstructing or generating 3D textured object meshes from a single image. The pipeline is built on the recent framework of multi-view diffusion combined with large reconstruction models (LRMs). If I understand correctly, the central claim of this paper is that "interpolated view synthesis using image diffusion models can provide more information and enhance 3D consistency and lead to improved object reconstruction results in LRMs." The motivation, as outlined in the paper, is that (1) Sparse image-to-multi-view models are not enough for LRM models, and (2) previous dense image-to-multi-view diffusion models (like video models) lack pixel-accurate 3D consistency, which can negatively impact the reconstruction quality of LRM models.

The authors specifically propose an infinite-view generation model that is conditioned on continuous camera poses rather than discrete ones. The images generated from these interpolated camera poses are then used as inputs to the LRM model to create a 3D mesh.

Overall, I find the paper's main points and motivations challenging to follow. From my experience, using dense views as input images does not benefit LRM models. Given this, it's unclear to me how interpolating between these inconsistent views would improve 3D consistency. I struggle to see how this approach addresses the core issue.
- As stated in the summary section, I do not think for the object reconstruction task, adding dense views for LRM models will help the reconstruction.
- The paper lacks in video results, or dense image results to show the quality of this “infinite” multi-view diffusion model. This part is crucial to help readers appreciate the whole method.
- The paper lacks quantitative ablation study on the view-interpolation design. There is only a qualitative comparison in Fig. 5, which is not convincing enough. Actually, I suspect the rationale of such multi-view interpolation design.
Yes
No ethics review needed.
As shown in the experiments, InfiniteMesh generates 3D textured meshes based on unnatural images. In contrast, if natural images are used, will the results still maintain good geometry consistency? Additionally, how does the proposed method perform in more complex cases?
It would be beneficial to present some ablation studies on the IVI module, particularly examining the design of separated reference and condition image.
1. The paper is well-written and easy to follow. The author has effectively organized the content and presented ideas in a clear and coherent manner.
2. Consistency in novel view synthesis is at the core of many image-to-3D task. The proposed method is able to outperform V3D and SV3D in multi-view consistency by a large margin. It seems that the introduced IVI module is efficient for generating more consistent meshes.
3. The paper demonstrates its effectiveness through substantial qualitative and quantitative experiments to verify its method outperforms baselines.
This paper presents an effcient image-to-3d generation method with improved multi-view consistency. Building on LRM and 2d diffusion model, it introduces an Infinite View Interpolation (IVI) module, designed to create two interpolated images between main views to improve the quality and consistency of the generated 3D mesh. A tri-plane-based mesh reconstruction is then used to convert these multi-view images into a final 3D mesh with superior texture and geometry. The method demonstrates great generation results.
1. The framework primarily builds on existing methods, incorporating previously developed components like the multi-view diffusion model Wonder3D and tri-plane-based Large Reconstruction Model (LRM). I expect the performance gain is great. However, this work does not show video results, and the results in Figure 3 is only marginally better than baselines. It is hard to validate the performance of this work.
2. simple comparison cases: The experimental comparisons mainly focus on simple cases. Including more complex and diverse cases would better showcase the model's strengths and highlight its potential advantages over competing methods.

I am open to further dicussion and willing to raise the score if the concerns are solved.
Yes
No ethics review needed.
1. When training the multi-view enhanced InstantMesh, are the input multiple views derived from the previous interpolation step or directly using ground truth ？
2. Have you considered performing secondary interpolation on the already interpolated views to obtain more viewpoint data to further improving the model's performance?
1. The paper's idea is easy to follow.
2. The proposed method demonstrates improvements in both accuracy and computational efficiency while enhancing the continuity and quality of the generated 3D contents.
This paper proposed an LRM-based image-to-3D generation with view interpolation. The proposed method consists of two key components: (1) an interpolation model that effectively expands 4 input views to 16 views by synthesizing intermediate viewpoints from multiview diffusion model outputs, and (2) a final LRM model that processes these interpolated views for 3D reconstruction. Experimental results demonstrate that this approach generates high-quality 3D content with superior texture and geometry, outperforming previous state-of-the-art methods.
1. The proposed method is merely a combination of existing methods, lacking novelty.
2. The model lacks evaluation of OOD (out-of-distribution) datasets or real-world datasets, making its generalization capability questionable.
3. The experimental settings are unclear in the comparative experiments. LGM is a model that converts 4 views to 3D Gaussians, yet it appears in Table 2 in single-image-to-3D comparison experiments. The authors need to further clarify the evaluation details of other models.
@inproceedings{
anonymous2024infinitemesh,
title={InfiniteMesh: View Interpolation using Multi-view Diffusion for 3D Mesh Reconstruction},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=pw0fKwi0Pj},
note={under review}
}
We present InfiniteMesh, a feed-forward framework for efficient high-quality image-to-3D generation with view interpolation. Recent advancements in Large Reconstruction Model (LRM) have demonstrated significant potential in extracting 3D content from multi-view images produced by 2D diffusion models. Nevertheless, challenges remain as 2D diffusion models often struggle to generate dense images with strong multi-view consistency, and LRMs often exacerbate this multi-view inconsistency during 3D reconstruction. To address these issues, we propose a novel framework based on LRM that employs 2D diffusion-based view interpolation to enhance the quality of the generated mesh. Leveraging multi-view images produced by a 2D diffusion model, our approach introduces an Infinite View Interpolation module to generate interpolated images from main views. Subsequently, we employ a tri-plane-based mesh reconstruction strategy to extract robust tokens from these multiple generated images and produce the final mesh. Extensive experiments indicate that our method generates high-quality 3D content in terms of both texture and geometry, surpassing previous state-of-the-art methods.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
View Interpolation Multi-view Diffusion 3D Mesh Reconstruction
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/7a2932224ddc3e0919d45f75207010dfe73896fe.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/163e9e4b2bdd49085a117aad41820705ca9184fd.pdf
InfiniteMesh: View Interpolation using Multi-view Diffusion for 3D Mesh Reconstruction
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission