Open-YOLO 3D: Towards Fast and Accurate Open-Vocabulary 3D Instance Segmentation
Open Vocabulary;3D point cloud instance segmentation
Yes
No ethics review needed.
1. In Table 1, does the class-agnostic Mask3D model have access to mask annotations for the same classes as those in the validation set? Do the other methods use the same class-agnostic segmentation model?

2. Since the proposed approach relies on a class-agnostic 3D instance generation model, what are the advantages of using only mask annotations, rather than both instance and label annotations, for training? I mean, are there practical scenarios where only mask annotations are available?

3. What does the tag "(Closed Vocab)" mean in Table 1? Does it indicate that the Mask3D method uses both mask annotations and object class annotations for training?

4. What is the performance of Mask3D (Closed Vocab.) on the Replica dataset?

Minor：

1. In line 092, a comma is missing after "multi-view information".
1. The paper is well-organized, and the ideas are clearly illustrated.
2. This paper introduces a novel approach for efficient open-vocabulary 3D instance labeling by leveraging 2D bounding box priors from a fast 2D object detector, demonstrating superior performance and time efficiency in experimental results.
3. A fast visibility computation algorithm (VAcc) is proposed to accelerate the process of associating 2D label maps with 3D proposals that may be occluded in some views.  This algorithm demonstrates both efficiency and robustness to variations in label map granularity.
This paper introduces an efficient 3D mask labeling method that leverages multi-view 2D label maps, referred to as Low Granularity (LG) Label Maps, created from 2D object bounding boxes to label 3D instances. The 3D instance (mask) proposals are generated using a pre-trained class-agnostic 3D segmentation method. To address object occlusion across different viewpoints, an Accelerated Visibility Computation (VACC) method is introduced, enabling rapid calculation of visibility matrices using intrinsic and extrinsic parameters.
1. The foundation of the proposed method is built upon the class-agnostic 3D segmentation model, Mask3D, which is used to generate 3D mask proposals. However, this paper lacks sufficient evidence to demonstrate Mask3D's effectiveness and generalizability for open vocabulary instance proposals.
2. The experimental evaluation of the proposed method for open-vocabulary 3D instance segmentation is relatively limited (only Table 6).
Yes
No ethics review needed.
See weakness.
1. **[Efficiency]** The experimental results suggest that the proposed method achieves high precision with a significantly better speed compared with most methods, and efficiency is important for practical deployment.

2. **[Clearness]** This paper explains its main contribution, how to assign class predictions to 3D proposals, with great clarity. The implementation details are elaborated sufficiently.
This paper primarily aims to achieve faster open-vocabulary 3D instance segmentation compared with existing methods like OpenMask3D. To realize this target, this work first uses a 3D instance segmentation network to generate segmentation proposals. Then, the output of an open-vocabulary 2D object detector as well as some designed 3D information is employed to derive the categories of these proposals.
1. **[Insufficient Academic Contributions]**: This work just combines the output of a 3D segmentation network and a well-implemented open-vocabulary 2D object detector to realize open-vocabulary 3D object detection (similar to existing open-world segmentation method, just with a replacement of the post network to 2D object detector), which is trivial. It is much faster than previous methods because previous methods are developed based on models like SAM and CLIP. This work employs more efficient and suitable existing models. Therefore, although this work is sound in terms of engineering, its real academic contribution and new insights are plain.

2. **[Insufficient Ablation Study]** As the method is efficient because it makes good use of existing models, it is important to clearly analyze how these models contribute to the efficiency, which will guide future works on how to develop an efficient open-vocabulary pipeline. However, this work fails to do so.

3. **[Misleading title]** The method name OPEN-YOLO 3D seems to be unsuitable. YOLO is a 2D object detector while the task is about 3D point cloud segmentation. Although the method utilizes the output of YOLO-World to generate class predictions, the method name is still a little misleading.
Yes
No ethics review needed.
Could you provide a more detailed theoretical analysis or complexity comparison of VAcc with the conventional method?
How does the method perform when the 2D object detector encounters difficult conditions, such as poor lighting or significant occlusion, if there's any evaluation of the robustness under such conditions?
What are the potential strategies for mitigating errors from misclassifications made by the 2D object detector, and how do they affect the 3D mask assignments?
I'm also particularly interested in the discussion in your limitation section, I was wondering would integrating fast 2D segmentation models, as mentioned, be feasible within your current framework? How might this affect both performance and speed?
This paper proposed a novel framework, which uses 2D object detection for 3D instance segmentation. The model they presented reduced computational overhead significantly. The Accelerated Visibility Computation (VAcc) leverages tensor operations and GPU batch processing, enabling highly parallelized visibility computation. This contributes to the following speed improvements without compromising performance. By integrating a high-performing 2D open-vocabulary detector, the framework retains strong zero-shot performance, which is important for real-world applications that use new or unknown object types.

It also includes detailed experiments that showcase Open-YOLO 3D's speech and accuracy, and highlight its performance above state-of-the-art approaches like Open3DIS and OpenMask3D. The paper also includes comprehensive ablation studies to demonstrate the improvement of each component. 

The overall writing is clear and the framework will be beneficial for related research.
This paper proposes Open-YOLO 3D, which is an open-vocabulary 3D instance segmentation framework that efficiently combines 2D object detection and 3D mask generation. The key idea of this paper is its reliance on bounding box predictions from a 2D open-vocabulary object detector and the subsequent use of these predictions for efficient 3D mask proposal and labeling. Unlike prior methods that use computationally intensive models such as SAM and CLIP for feature lifting from 2D to 3D, this paper uses a novel Multi-View Prompt Distribution (MVPDist) and Accelerated Visibility Computation (VAcc) methods to speed up the segmentation process. The framework this paper proposed achieves up to 16x faster inference while keeping competitive or better accuracy.
I like the overall framework this paper presents and appreciate its contribution to 3D instance segmentation by introducing an inference-efficient model, but I still have some concerns about it:

While the paper mentions that VAcc uses tensor operations, a deeper explanation or complexity analysis comparing it to conventional iterative methods would strengthen the understanding of its true computational advantage, and the reason why it can achieve faster inference speed. I believe the paper clearly demonstrates the operation of this proposed algorithm, however, more explanation about why it is efficient and how much computation cost it saves will better demonstrate the paper's contribution.

The method relies on the quality of the 2D object detector, and this might be an issue if the 2D views are suboptimal (for example poor lighting, and occlusions). A more extensive analysis or discussion on how 2D detection failures propagate through the pipeline would add value.
Yes
No ethics review needed.
Please see the weakness.
1.	The authors only utilize the bounding boxes from 2D object detectors to alleviate the redundancy brought by 2D masks, which demonstrates a significant improvement in inference speed compared to OpenMask3D.
2.	The authors propose a Multi-View Prompt Distribution to obtain reliable category labels form 3D masks, the experimental results evaluation on the ScanNet200 and Replica datasets prove the efficiency of the method proposed by the authors.
3.	The paper is well-structured, and the connection between the proposed method and the motivation is coincident.
The authors propose an efficient method for open-vocabulary 3D instance segmentation to enhance the real-time capability. Unlike existing methods that rely on obtaining 2D masks and category labels from 2D foundation models (like SAM and CLIP), the authors propose a novel approach, Open-YOLO3D, which only leverages bounding boxes generated by 2D object detectors. Moreover, the authors propose a Multi-View Prompt Distribution (MVPDist) method to endeavor promising performance in recognition. The experimental results demonstrate the promising real-time performance of the method proposed by the authors.
1.	The improvements of the segmentation performance observed in Open-YOLO3D primarily arise from the enhanced category recognition, which is likely from the prior knowledge of the pre-trained YoloWorld model.
2.	The challenges inherent in Open-YOLO3D closely resemble those faced by Open3DIS, as both methods rely on pre-trained models for generating 3D proposals. As discussed in Open3DIS, the pre-trained 3D models have limited capabilities when it comes to detecting uncommon categories. The representation of 3D data for open vocabulary instance segmentation might be uncultivated and limited.
3.	Recent studies [1] have indicated that OpenMask3D performs poorly on certain outdoor datasets, such as NuScenes. Does Open-YOLO3D face similar challenges in effectively identifying sparse-diverse and less common categories in outdoor environments?

[1] Open-Vocabulary SAM3D: Towards Training-free Open-Vocabulary 3D Scene Understanding.
Yes
No ethics review needed.
Please draw my concerns listed in the Weaknesses part.
1.	The proposed method is simple yet effective.
2.	The two proposed designs are helpful for 3D instance segmentation with meaningful designs.
3.	The experimental results show that the proposed method could achieve good performance while remaining very efficient.
This paper aims to deal with the open-vocabulary 3D instance segmentation task with a fast and cost-effective approach by utilizing a YOLO-style design. A Multi-View Prompt Distribution method is proposed to effectively fuse the multi-view information. The low granularity label maps are proposed to only use 2D detectors to predict prompt IDs for 3D instance masks. Experimental results demonstrate the state-of-the-art performance of the proposed method. The speed of the proposed method is about 16 times faster than that of existing methods.
1.	Is it possible to extend the proposed method on panoptic segmentation of 3D scenarios? Please present your design briefly for this. 
2.	As shown in Table 1, the inference time of the proposed method is 21.8, which is slower than OpenScene (3D Distill). Please add the explanation for this phenomenon in the corresponding text (first paragraph of Section 5.1).
3.	In Line 405, it should be 4.29 but not 04.29 for OpenScene (3D Distill).
@inproceedings{
anonymous2024openyolo,
title={Open-{YOLO} 3D: Towards Fast and Accurate Open-Vocabulary 3D Instance Segmentation},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=CRmiX0v16e},
note={under review}
}
Recent works on open-vocabulary 3D instance segmentation show strong promise but at the cost of slow inference speed and high computation requirements. This high computation cost is typically due to their heavy reliance on aggregated clip features from multi-view, which require computationally expensive 2D foundation models like Segment Anything (SAM) and CLIP. Consequently, this hampers their applicability in many real-world applications that require both fast and accurate predictions. To this end, we propose a novel open-vocabulary 3D instance segmentation approach, named Open-YOLO 3D, that efficiently leverages only 2D object detection from multi-view RGB images for open-vocabulary 3D instance segmentation. 
 We demonstrate that our proposed Multi-View Prompt Distribution (MVPDist) method makes use of multi-view information to account for misclassification from the object detector to predict a reliable label for 3D instance masks. Furthermore, since projections of 3D object instances are already contained within the 2D bounding boxes, we show that our proposed low granularity label maps, which require only a 2D object detector to construct, are sufficient and very fast to predict prompt IDs for 3D instance masks when used with our proposed MVPDist.
 We validate our Open-YOLO 3D on two benchmarks, ScanNet200 and Replica, 
 under two scenarios: (i) with ground truth masks, where labels are required for given object proposals, and (ii) with class-agnostic 3D proposals generated from a 3D proposal network.
 Our Open-YOLO 3D achieves state-of-the-art performance on both datasets while obtaining up to $\sim$16$\times$ speedup compared to the best existing method in literature. On ScanNet200 val. set, our Open-YOLO 3D achieves mean average precision (mAP) of 24.7% while operating at 22 seconds per scene. Our code will be publically available.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Open Vocabulary 3D point cloud instance segmentation
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/b69d5286d69fe6fd69c5ad602fd6b16102210337.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/30717cdf372858d9e8cdda4e3545dafcff7b109d.pdf
Open-YOLO 3D: Towards Fast and Accurate Open-Vocabulary 3D Instance Segmentation
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission