CityAnchor: City-scale 3D Visual Grounding with Multi-modality LLMs
3D Visual Grounding;Large language model;multi-modality language model
Yes
No ethics review needed.
Please address the questions in the weaknesses section.
1. The paper is well-written and easy to follow.
2. The experiment results are impressive.
3. The proposed method is well motivated and technically sound.
This paper proposes a multi-modal LLM for city-scale 3D visual grounding named CityAnchor. CityAnchor adopts a coarse-to-fine searching strategy. First, a 3D segmentation model generates the potential objects from point clouds. Next, in the coarse localization stage, a LLaVA generates the <RoI> token and a SAM generates the RoI heapmap indicating the candidates of the target object. At last, in the fine-grained matching stage, another LLaVA select the best candidate as the target object. Besides, this work also presents a new dataset for 3D visual grounding. Experiments show that CityAnchor outperforms the previous methods by a large margin.
1. As also mentioned in Sec.5, the efficiency of the proposed method is a major concern. It involves two LLMs and multiple forward passes in the fine-grained matching stage. Are there any possible solutions to improve the efficiency?
2. How to choose the positive and the negative samples during the training of FMM?
3. The objects are generated by a pretrained 3D segmentation model which can only recognize a closed set of targets. So I expect to see the generality of CityAnchor to unknown objects.
4. Some important details are missing:
- Line 157, what is $T_c$?
- Line 138 & 209, how to determine whether an object has a landmark name?
- Line 206, what is $c$ in $E^s_x$?
- Line 232, which encoder is the concatenated feature vector fed into?
Yes
No ethics review needed.
I mentioned all comments including reasons and suggestions in the above sections. I recommend that the author will provide all the concerns, and improve the completeness of the paper. If the rebuttal period resolves the above-mentioned concerns, I will gladly raise my score. Also, there are little vague sentences and grammatical errors in the paper. I recommend that the author will revise the paper.
First of all, it is interesting that a highly intuitive idea is proposed to address two major issues in existing methods concerning how to improve the multi-modality feature extraction in the large-scale visual grounding and how to efficiently localize the object in a large-scale point cloud.

Although it lacks technical novelty, it effectively implements the necessary modules to address two main problems. More specifically, a coarse location firstly finds possible regions on a projected 2D map of the point cloud while the fine-grained matching accurately determines the most similar region with the given test description.

In experimental section, the proposed method achieved SoTA results in city-level localization tasks, demonstrating its effectiveness. The ablation study is highly analytical for each level module and feature representation.
In the paper, the authors introduced CityAnchor, a 3D visual grounding method tailored for city-scale scenes. The CityAnchor is based on a two-stage multi-modal LLM. In the coarse stage, CityAnchor predicts candidate objects that match the query text descriptions on projected 2D maps derived from urban point clouds, allowing us to efficiently filter out redundant regions and concentrate on likely objects. Next, it performs fine-grained matching between text descriptions and the candidate objects to establish the final grounding results by encoding both text descriptions and object features into an LLM. To do that, the authors created a new dataset and evaluated the proposed method on both CityAnchor and CityRefer datasets to demonstrate the effectiveness of the proposed method on city-scale point clouds.
One concern is the validity of the coarse localization module. As shown Fig.4, I am curious whether CLM operates as intended. It appears to activate too many candidates, as if it were merely distinguishing between urban and non-urban areas. This raises doubts about whether it significantly aids in the efficient operating in the overall framework.

Experiment analysis and description are not specific and descriptive. For example, a detailed explanation based on specific examples is needed to clarify the novel objects and novel descriptions, whether the reason they are not in the training data is due to an out-of-distribution (OOD) scenario.
Yes
No ethics review needed.
It’s unclear how the method addresses dynamic objects in urban environments, such as moving vehicles or pedestrians, which could affect the reliability of grounding results.
S1: The method’s coarse-to-fine design is well-structured for efficient large-scale 3D visual grounding.

S2: Experimental results show substantial and objective improvements over existing methods.

S3: The proposed model demonstrates clear practical potential for applications in urban planning and geospatial analysis.
The paper addresses the challenge of object localization in city-scale 3D point clouds with a method called CityAnchor, which leverages a multi-modal large language model. Using a two-stage approach of coarse localization and fine-grained matching, CityAnchor efficiently and accurately identifies target objects in large urban scenes. Experimental results show that CityAnchor significantly outperforms existing methods in accuracy and speed, with promising applications in urban planning and geospatial analysis.
W1: The method seems heavily reliant on large language models, and the need for manual adjustment of the Region of Interest (RoI) threshold in the coarse localization stage suggests a lack of generalizability.

W2: While the coarse-to-fine localization design is effective, it builds on existing multi-stage approaches without introducing fundamentally new concepts in 3D visual grounding, which suggests that the method's strengths lack a degree of innovation.

Other:
The layout could be improved; for instance, all teaser images should be embedded as PDFs rather than PNGs to ensure the text within images is selectable.
Yes
No ethics review needed.
- Effectiveness of CLM: In Figure 4, it appears that CLM is not highly effective, as it predominantly highlights houses and roads. Could the authors provide additional results or offer an explanation for this behavior?

- Prompt Engineering Requirements: It seems that users must invest effort in engineering their prompts to achieve accurate results, which may pose usability challenges. The authors could consider conducting a user study to gather prompts from a diverse set of users to assess the method’s robustness when different descriptions target the same object.

- Progressive Prompting Approach: It would be intuitive to structure prompts in a coarse-to-fine manner—beginning with a broad query that identifies multiple potential candidates and gradually introducing constraints to precisely locate the target.
+ The paper introduces CityAnchor, a novel approach to 3D visual grounding specifically tailored for city-scale scenes. Its use of a two-stage multi-modality large language model (LLM) for both coarse localization and fine-grained matching is effective and advance in the field. The proposed approach effectively integrates spatial context and object features, showcases the ability in addressing the complexities of urban environments, setting it apart from existing methods that may not adequately handle such challenges.
+ The paper provides a comprehensive evaluation of the model's performance, demonstrating significant improvements in accuracy compared to existing methods. Additionally, the introduction of the CityAnchor dataset as a synthesized benchmark for 3D visual grounding further enhances the quality of the research, providing a valuable resource for future studies.
+ The paper is well-structured and easy to follow, clearly stating the objectives, methodology, and results of the research.
The paper presents CityAnchor, a novel 3D visual grounding method designed for city-scale scenes, utilizing a two-stage multi-modality large language model (LLM). The first stage focuses on coarse localization of potential object regions based on text descriptions, while the second stage involves fine-grained matching to determine the best object matches. Specifically, 
- Coarse Localization Module (CLM)  takes a city-scale colored point cloud, projects it onto a 2D map, and uses the text description to regress possible regions of the target on this 2D map.
- Fine-grained Matching Module (FMM) performs fine-grained comparisons between each candidate object identified by the CLM and the text description by predictiong the similarity between the text and each candidate object, and select the object with the highest similarity score.
Additionally, the paper introduces the CityAnchor dataset, a synthesized benchmark for 3D visual grounding. The method demonstrates significant improvements in accuracy and efficiency over existing techniques on the CityRefer and CityAnchor datasets.
- Limitations of 2D Mapping: Projecting 3D space onto a 2D map often results in a loss of spatial information, particularly in cases of occlusions or overlapping instances. This pipeline, especially the CLM component, operates under the significant assumption that all objects are nearly flat, which inherently limits its applicability to more complex environments, such as indoor spaces or densely populated urban areas like NYC.

- Dependence on Point Segmentation Model: The pipeline requires users to first segment the scene point cloud into objects using a pretrained 3D segmentation model. As a result, it heavily depends on the granularity and accuracy of this pretrained model, setting an upper performance limit for the proposed approach.

- Justification for the Proposed Dataset: While introducing a new dataset is valuable, the necessity of the CityAnchor dataset for this task or the proposed method requires further clarification. It is essential to address whether the method’s performance would significantly degrade without this dataset, whether it enhances diversity or robustness, and if it offers any unique attributes that the CityRefer dataset lacks. Providing these details would strengthen the contribution of the dataset to the paper.

- Inference time and hyperparameter tuning: These two limitations were briefly discussed in the paper.
We present a city-scale 3D visual grounding system to accurately find targets in point clouds from text descriptions.
@inproceedings{
anonymous2024cityanchor,
title={CityAnchor: City-scale 3D Visual Grounding with Multi-modality {LLM}s},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=7nOl5W6xU4},
note={under review}
}
In this paper, we present a 3D visual grounding method called CityAnchor for localizing an urban object in a city-scale point cloud. Recent developments in multiview reconstruction enable us to reconstruct city-scale point clouds but how to conduct visual grounding on such a large-scale urban point cloud remains an open problem. Previous 3D visual grounding system mainly concentrates on localizing an object in an image or a small-scale point cloud, which is not accurate and efficient enough to scale up to a city-scale point cloud. We address this problem with a multi-modality LLM which consists of two stages, a coarse localization and a fine-grained matching. Given the text descriptions, the coarse localization stage locates possible regions on a projected 2D map of the point cloud while the fine-grained matching stage accurately determines the most matched object in these possible regions. We conduct experiments on the CityRefer dataset and a new synthetic dataset annotated by us, both of which demonstrate our method can produce accurate 3D visual grounding on a city-scale 3D point cloud.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
3D Visual Grounding Large language model multi-modality language model
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/9e70d26a58b502ac36bbe784bc364763f01cf1f2.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
CityAnchor: City-scale 3D Visual Grounding with Multi-modality LLMs
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission