NeuralPlane: Structured 3D Reconstruction in Planar Primitives with Neural Fields
3D Reconstruction;3D Scene Understanding;Scene Abstraction;Neural Rendering
Yes
No ethics review needed.
I am impressed by the technical contribution made by authors for this work. However, there also exist a few major concerns for me at this time which discourage me to grant this paper a higher value. 

Please try to address my concerns listed in the weakness part. I will accordingly consider to improve my overall rating if the concerns are well solved.
1. The unique advantage of the system is the association of geometry and semantic features for plane reconstruction problem, which requires both geometry-aware perception and semantic-aware grouping.
2. The paper leverages foundation models to achieve plane segment initializaing and utilize geometry-driven losses to optimize the system. There is no groundtruth plane segmentation or geometry label required. 
3. Overall the paper and diagrams are well written and presented.
4. The experimental comparison and thorough and convincing on both plane geometry reconstruction and segmentation.
This paper presents a neural 3D reconstruction system on 3D plane reconstruction of indoor scenes. Inspired by the recent success of neural radiance field and image foundation models (SAM2), this paper presents a multi-view 3D reconstruction pipeline leveraging these techniques. The training scheme consists of three phases: Initializing plane segments and parameters -> optimizing a neural feature field for plane-specific feature representation (encouraged by a list of geometry-guided loses) -> plane extraction by grouped features and RANSAC. Experiments are conducted extensively on two representative indoor datasets: ScanNet and 7-scenes and are compared with a group of competitve baseline methods.
1. I think the major advantage of this paper is the unsupervised learning paradigm on the plane reconstruction problem. However, since both ScanNet and ScanNet++ has groundtruth images, this unique advantages seems not to be fully enjoyed and reflected. So I suggest authors to apply the proposed system on some outdoor scenes containing plane structures (such as autonomous driving dataset or street view datasets), to verify the adaptability of the method.
 
2. I have tested PlaneRecon in my previous projects. It can incrementally reconstructs planes in an online and real-time manner. Besides, it can trains over multiple scenes and directly test without any test-time optimization. Although its precision should fall behind than the neural reconstruction papers, the generalizability and speed is higher than the proposed method. I am curious on whether the paper has such potential to tackle these limitations.

3. Missing a few related works: (1) Recovering 3d planes from a single image via convolutional neural networks (2) PlaneMVS: 3D Plane Reconstruction From Multi-View Stereo (3) Single-image piece-wise planar 3d reconstruction via associative embedding.

4. On quantitative evaluation, it is unclear that what is the groundtruth is here. Does it stand for the groundtruth planar part only or the entire mesh? As a plane reconstruction paper, I think the former one should be more reasonable. If so, the first several methods listed in Table 1 cannot directly be compared with the proposed method. Authors should make it clearer on this part.

5. Is the proposed method robust to the hyperparameters listed in the paper especially (1) to,tn and m in the push loss during training and (2) the parameters selected in RANSAC during plane fitting? Some ablation studies on hyper-parameter robustness are expected to make the method more generalizable across most indoor scenes, since the geometric scale and semantic distribution can have large variance among scenes.
Yes
No ethics review needed.
Since it is a complex system paper, making it hard to reproduce, will the code be publicly avaliable?
- This paper is well-written and easy to understand.
- The proposed method does not require ground-truth plane annotations, as it can learn effectively from noisy monocular model outputs.
- The method demonstrates SOTA performance and achieves clean plane segmentation results.
This paper presents NeuralPlane, a method for reconstructing 3D scene plane primitives via neural fields without GT plane labelling. The method is divided into three main stages: firstly, it combines pre-trained normal prediction and SAM to generate initial 2D planar segments and estimates their 3D parameters using SfM keypoints. Secondly, it optimises two neural fields, a density field based on planar geometric constraints, and a coplanar neural field that understands the semantic relationships between regions . The neural coplanar field is followed by a neural parser module that helps to model the learned coplanar relations. Finally, the optimised neural representations are converted into explicit 3D planes through point sampling, feature-based clustering and RANSAC fitting. The method is evaluated on the ScanNetv2 and ScanNet++ datasets, and the results show that it outperforms both the learning-based method and the Geometry+RANSAC method in terms of geometric and semantic metrics.
- The proposed method involves numerous hyperparameters, including balancing parameters for loss, the number of semantic prototypes, and parameters listed in Lines 850-863.
- As a complex system, it is important to discuss and present failure cases to help readers understand the method’s limitations.
Yes
/
No ethics review needed.
1. Eqn 4: n_i is not defined, is it the surface normal of the selected local planar primitive? Please explicitly define n_i in the text or equation, and confirm if it refers to the surface normal of the local planar primitive.
1. Novelty in Combining Geometry and Semantics: The method’s approach to merging geometry with semantic information through a neural coplanarity field is innovative, enhancing the semantic consistency of 3D reconstructions. A complex system with multiple stages is proposed to estimate/reason the local planar regions and the associated parameters, to associate the planes in 3D using radiance field, to resolve semantic conflicts using Neural Coplanarity Field.
2. High-Quality Reconstruction: Experimental results indicate that NeuralPlane achieves fine-grained and coherent plane reconstructions, outperforming existing methods in most of the metrics. 
3. Efficiency: NeuralPlane’s volume density representation allows for faster training compared to implicit methods, an important practical advantage.
4. Extensive ablation studies: the authors include sufficient ablation studies to support the effectiveness of the proposed modules.
5. Good writing: the paper is clearly written that the technical details are clearly presented.
The paper presents NeuralPlane, a novel approach to 3D plane reconstruction that utilizes neural fields for generating structured 3D maps from multi-view images without the need for plane annotations. The method emphasizes two main aspects: geometry and semantics. Key contributions include:
1. Monocular Plane Segmentation: A monocular module extracts geometrically smooth (based on off-the-shelves Surface Normal Predictor) and semantically meaningful 2D plane observations (based on Segmenet Anything Model). 
2. Plane-Guided Neural Representation: The model utilizes these 2D segments to train a neural field that captures accurate 3D plane locations. A surface normal regularization and pseudo-depth regularization terms are proposed.
3. Neural Coplanarity Field: This self-supervised feature field enables semantic consistency within the 3D reconstruction by grouping planar regions that share coplanar relationships. A contrastive loss is proposed to distinguish between planes with similar geometric properties but different semantic properties. 
The method demonstrates superior performance on ScanNetv2 and ScanNet++ datasets, indicating its effectiveness in indoor environments.
1. Complexity of Methodology: The proposed method involves several stages, including monocular plane segmentation, neural coplanarity field training, and plane extraction. In each stage, there are several submodules and I believe there are some hyperparameters decisions in each stage. While effective, this complexity (especially the combination of submodules and associated parameters) may impact its scalability to larger scenes or generalizability. For example, K-means clustering on predicted normal map, mask size threshold in SAM, thresholds to form negative pairs in Neural Coplanarity Field, Loss balancing parameters, etc. It would be great if the authors could share the insights on the impact of these hyperparameter settings on different scenes. For example, is a universal set of hyperparameters applied to all the test scenes? It would be great if the authors could provide a sensitivity analysis or ablation study on key hyperparameters across different scenes. This would help clarify how robust the method is to parameter changes and whether a universal set of parameters is feasible.

2. Dependence on Initial 2D Plane Segmentation Quality: As the regularizations are based on the quality of initial local plane geoemtry, the method’s success can depend on the quality of 2D plane segments obtained from monocular priors, which may introduce inaccuracies in challenging environments for monocular predictors. As the authors mentioned , the local planar primitives can result in severe inconsistency across views (Line 194). 

3. Over-Segmentation Issue: As highlighted in the paper, the Segment Anything Model (SAM) tends to over-segment planes, resulting in multiple smaller plane segments for a single surface. Although this is managed in the training process, it may require further refinement to avoid segmentation inconsistencies in complex scenes. From the visualization on the GitHub page, it seems to me that some of the small segments usually resulted in inaccuracy.
Yes
No ethics review needed.
* What's the computation complexity of this work? Like the GPU memory usage and training time? How's it compared to related works?
This paper proposed a comprehensive 3D reconstruction framework with plane primitives. 

* Compared to some of the similar works that only focus on detecting the planes, this work also utilizes the detected planes to guide neural field learning.

* The presentation of this paper is clear and concise. 

* The qualitative and quantitative results look promising.
This paper proposed a framework called NeuralPlane to reconstruct 3D indoor scenes as planar primitives from posed 2D images. The author first employs 2D prior models to generate local planar primitives, then uses the geometric and semantic priors to guide the NeRF-style reconstruction learning. Finally, a decoding algorithm is designed to extract the global explicit plane mesh from the learned neural field.
Extensive experiments are conducted to evaluate the performance on ScanNetv2 and ScanNet++ datasets.
* This work involves a lot of submodules, especially the neural field with geometric, semantic, and coplanar features could be computationally expensive.

* A concurrent work is very relevant to this paper and could be discussed in the related works section:
Chen, Zheng, et al. "PlanarNeRF: Online Learning of Planar Primitives with Neural Radiance Fields." arXiv preprint arXiv:2401.00871 (2023).

* The text illustration and conceptual figure of Neural Parser could be improved, current version is not clear enough and easy to follow.
NeuralPlane rebuilds indoor scenes as arrangements of planar primitives from multi-view images.
@inproceedings{
anonymous2024neuralplane,
title={NeuralPlane: Structured 3D Reconstruction in Planar Primitives with Neural Fields},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=5UKrnKuspb},
note={under review}
}
3D maps assembled from planar primitives are compact and expressive in representing man-made environments, making them suitable for a spectrum of applications. In this paper, we present **NeuralPlane**, a novel approach that explores **neural** fields for multi-view 3D **plane** reconstruction. Our method is centered upon the core idea of distilling geometric and semantic cues from inconsistent 2D plane observations into a unified 3D neural representation, which unlocks the full leverage of plane attributes. This idea is accomplished by NeuralPlane via several key designs, including: 1) a monocular module that generates geometrically smooth and semantically meaningful segments as 2D plane observations, 2) a plane-guided training procedure that implicitly learns accurate plane locations from multi-view plane observations, and 3) a self-supervised feature field termed *Neural Coplanarity Field* that enables the modeling of scene semantics alongside the geometry. Without relying on plane annotations, our method achieves high-fidelity reconstruction comprising planar primitives that are not only crisp but also well-aligned with the semantic content. Comprehensive experiments on ScanNetv2 and ScanNet++ demonstrate the superiority of our results in both geometry and semantics.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
3D Reconstruction 3D Scene Understanding Scene Abstraction Neural Rendering
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/f70e3a54efbb9df8d1ba24ab403fd09f6d6365f6.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/36ba4604cc4e785b5fa76f17b85f39b190e4fdea.zip
NeuralPlane: Structured 3D Reconstruction in Planar Primitives with Neural Fields
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission