SPARTUN3D: Situated Spatial Understanding of 3D World in Large Language Model
Situated Understanding in 3D Scen;3D VL;LLM
Yes
No ethics review needed.
Please refer to the weakness section.
1. The situated scene graph is well-crafted and contributes to a higher-quality dataset that supports future research in situated 3D scene understanding.
2. The paper is well-written and easy to follow.
This paper introduces Spartun3D, a scalable dataset designed to enhance situated 3D understanding. The authors construct a situated scene graph to facilitate the generation of situated captions and QA pairs via LLM prompting. Additionally, they incorporate a situated spatial alignment module into a baseline 3D LLM to improve scene understanding. Experimental results demonstrate the effectiveness of their approach.
1. The paper could benefit from including more examples of generated data, particularly those with errors, to provide better insight into the dataset's quality. It would be valuable to discuss potential methods for correcting inaccurate captions or QA pairs generated by the LLM, as well as the impact of these errors on model performance.
2. There is a lack of a detailed ablation study on the proposed modules, specifically the Situated Textual Description and 3D Object-Text Alignment, which would help clarify their individual contributions.
3. The evaluation of object navigation relies on only four simple actions, which may weaken the findings, although it does showcase some zero-shot capabilities of the model.
Yes
No ethics review needed.
- Some additional discussion / proof on why the additional automated data is useful. Scaling curves with varying amount of Spartan3D-LLM data used in training would help -- on real world benchmarks like SQA3D.

- Clarification on the newly constructed dataset from 3RScan for zero-shot LEO baselines, and generalization capabilities of the proposed model

- More details on the navigation experiments in Table-5; specifically regarding the training datasets used for the proposed model and the baselines.
- The paper is very well written and clearly lays down the problem and the proposed solution (with informative figures)
- The Spartan3D dataset will be useful to the community
- The proposed method obtains better results than prior state of the art on SQA3D and the proposed explicit alignment loss helps performance in all tasks (by about 1-2%)
The paper addresses the task of situated reasoning where an agent is located at a certain location in the 3D space and needs to reason about the situation (caption or answer questions) from it's spatial location. The paper generates a new dataset called Spartan3D, relying on GPT-4o that lets them scale the size of the data; and proposes an explicit spatial alignment module where the object tokens are aligned to a desciption of spatial situations/relations of the corresponding object obtained via a template. The method is tested on situation question answering benchmark of SQA3D and the newly proposed Spartan3D dataset. The results show that the explicit spatial alignment module helps in question answering as well as other captioning tasks; and the additional Spartan3D dataset helps performance on all datasets.
- A big claim of the paper is that SQA3D is human collected dataset; and the proposed pipeline to generate SPARTUN3D can be very useful to scale up the dataset size. However, it is unclear how effective this automatically generated dataset is in comparison to human generated dataset. Specifically, the current experiments of Table-3 show that on real-world SQA3D benchmarks, using additional Spartan3D dataset on top of human collected SQA3D dataset helps performance by about 1-2% despite much larger dataset size. Additionally, just training on Spartan3D dataset results in significantly worse performance on SQA3D (by about 20%). Moreover, it is unclear if the automated data shows strong scaling behaviors -- which, if true, would push the community to scale up the datasets in a similar way that this paper proposed instead of collecting more human annotations.

 - L460-465 "In the zeroshot setting, we re-trained LEO exclusively on a dataset constructed from 3RScan to ensure a fair
comparison with our method. As shown in Table 3, LEO performs poorly on SQA3D in the zeroshot setting, suggesting its limitations in learning situated understanding from its original dataset. In contrast, LEO trained on Spartun3D shows significant improvement, demonstrating the effectiveness of our dataset and the generalization ability of our model."

The above set of lines are confusing to me. 
- What exactly is this dataset constructed from 3RScan which is constructed for the zero-shot LEO baseline? Without this information, currently the conclusion seems to be that Spartan3D dataset is better than some other way of constructing a training dataset.
- I did not follow how obtaining good performance from LEO trained on Spartun3D leads to the "generalization ability of our model" conclusion? Table-3 does not show how the proposed model works with the "newly constructed 3RScan data" instead of the Spartan3D dataset. 

- For navigation experiments in Table-5. what is Spartan3D-LLM trained on? Is LEO and Spartan3D-LLM identical except the explicit spatial alignment module for this experiment? I am trying to understand the main reason why LEO does not work at all for navigation while Spartan3D-LLM show some non-zero performance.
Yes
No ethics review needed.
1) By using GPT4 API, how to ensure the quality and accuracy of datasets? Can you provide more information on quality control measures for SPARTUN3D? Specifically, how are errors from the automated pipeline identified and handled?
2) How does the Situated Spatial Alignment Module impact computational resources, particularly in training time and memory usage? Would the module be feasible for real-time applications?
3) Have you considered evaluating Spartun3D-LLM on other embodied tasks beyond QA and navigation, such as robotic manipulation, multi-step reasoning in dynamically changing environments or some other 3D tasks like visual grounding, 3D object detection or dense caption?
4) Your chosen baselines like Leo,3D-Vista and 3D-LLM, were relesed in 2023,  the paper lacks a broader comparison with recent advances, please show some comparisons of other advances of 3D-LLMs with SPARTUN3D-LLM?
5) As the alignment is one of your contributions of this paepr. Are there other alignment strategies that were considered or experimented with?
1) Innovative Dataset: SPARTUN3D addresses a key limitation in 3D-based LLMs by providing an extensive dataset that incorporates situated reasoning. This dataset, generated by leveraging GPT-4o, is both scalable and diverse, which is essential for training models in 3D scene understanding from dynamic perspectives. This is a well-motivated and relevant contribution to the field.

2) Effective Alignment Module: The Situated Spatial Alignment Module enhances model performance by aligning 3D visual representations with their textual counterparts. This module appears to be a critical contribution as it enables more accurate spatial understanding and has implications for embodied tasks. The use of spatial self-attention and a 3D object-text alignment loss to improve object-text coherence is particularly novel.

3) Comprehensive Experiments: The experimental evaluation is thorough, covering both in-domain tasks (Spartun3D and SQA3D) and out-of-domain tasks (MP3D ObjNav), with significant improvements shown in zero-shot navigation accuracy. Ablation studies and visualizations further confirm the efficacy of the alignment module in improving spatial reasoning capabilities.

4) Practical Applications: Situated spatial understanding is valuable for real-world applications, such as navigation and human-robot interaction. This paper makes a strong case for the applicability of SPARTUN3D in these areas.
This paper introduces SPARTUN3D, a dataset aimed at enhancing the spatial reasoning abilities of 3D-based large language models (LLMs) by providing situated context. The dataset includes tasks like situated captioning and QA, challenging models to respond based on the agent's dynamic perspective. Additionally, the authors propose a novel alignment module in Spartun3D-LLM, which improves alignment between 3D scene representations and textual descriptions. The paper demonstrates the model’s generalization on multiple 3D tasks, outperforming baselines in spatial understanding and navigation accuracy.
1) The SPARTUN3D dataset is automatically generated with GPT-4o, which, while scalable, may lack the nuanced spatial and contextual fidelity found in human-annotated data. This synthetic nature raises questions about the dataset’s generalizability and its ability to represent real-world scenarios accurately.

2) The evaluation primarily focuses on situated QA, captioning, and navigation tasks, which, although useful, may not fully capture the complexity of embodied tasks (e.g., robotic manipulation or multi-step planning). Expanding the evaluation to include a wider variety of tasks would provide a more comprehensive assessment of the model's situated understanding abilities.

3) While the Situated Spatial Alignment Module is an interesting addition, the paper lacks a rigorous theoretical foundation for its design. Details on why specific techniques (e.g., spatial self-attention and MSE for alignment) were chosen and how they uniquely contribute to spatial alignment are not thoroughly explained, which could weaken the perception of this module’s novelty. Also, the added complexity of the alignment module may lead to increased computational demands, yet the paper does not discuss or benchmark these costs. For practical deployment in real-time applications or on resource-limited systems, it is essential to understand the trade-offs between the module’s benefits and its computational impact.

4) While comparisons to the LEO (2023.11 released) baseline and other similar models are present, the paper lacks a broader comparison with recent advances in spatial reasoning, navigation models or 3D tasks.
Yes
No ethics review needed.
(1). The input to the model is triple <C, S, Q>, where C is the 3D scene context, S is the situation, and Q is a question. The situation S can be further denoted as $S^t$, $S^p$, $S^r$, where $S^p$ is the 3D coordinate in the form <x, y, z> and $S^r$ is the quaternion. In the dataset, line 182, authors assume the agent's orientation is always facing forward to the center of the selected object. How is this selected object chosen? Will this introduce any bias?

(2). Can author show how they make inference of SPARTUN3D? More specifically, what is S, C, and Q respectively? Is it possible to provide an example?

(3). Why does finetuned 3D-LLM only has EM score? I understand 3D-Vista is a BERT but 3D-LLM seems not?
(1). This paper is very well-motivated. The importance of situational reasoning in 3D-LLM is beyond doubt.

(2). The writing of the paper is sound. 

(3). This paper covers a range of tasks that are less explored by 3D-LLMs.
SPARTUN3D addresses key limitations in current 3D LLMs:

- Existing 3D datasets are lack of situated context
- Current 3D LLM's architecture lack explicit alignment between the spatial representations of 3D scenes and natural language.

To tackle these issues, this paper introduces a novel dataset, SPARTUN3D, along with a model enhancement named SPARTUN3D-LLM, featuring a dedicated spatial alignment module. Experimental results validate that both the dataset and model improvements significantly advance the situated spatial reasoning capabilities of 3D LLMs.
(1). The motivation behind this dataset is strong, and I acknowledge the considerable effort put into curating it. However, the dataset’s quality remains unverified. While the authors took steps to avoid unnatural configurations (see line 176), I am not yet convinced of its robustness. Specifically, the input scene graph includes angular relationships calculated using the centroids of three objects. Given the variability in object sizes, it is uncertain whether GPT-4o can accurately interpret and annotate these spatial relations. What evidence supports the notion that these spatial relationships are meaningful to GPT-4o, and do these annotations align with human perception?

(2). GPT-4o’s capacity to generate large volumes of data makes it ideal for annotating 3D scenes at scale. However, the paper lacks scaling experiments to demonstrate how model performance improves as more data is generated using this pipeline. Including such experiments would provide valuable insight into the benefits of scaling for this approach.

(3). The input to the model is triple <C, S, Q>, where C is the 3D scene context, S is the situation, and Q is a question. The situation S can be further denoted as $S^t$, $S^p$, $S^r$, where $S^p$ is the 3D coordinate in the form <x, y, z> and $S^r$ is the quaternion. Can author provide more analysis on how <x, y, z> and $S^r$ would affect models' performance?

(4). The setting is egocentric situational 3D-LLM, but SPARTUN3D takes mostly text information as input. Is it possible to directly takes a 2D image as egocentric condition?

(5). For the experiments Navigation Performance in zero-shot manner, LEO's training data is very simple. It will be interesting to see a comparison between finetuned LEO with SPARTUN3D and Sparttun3D-LLM.
@inproceedings{
anonymous2024spartund,
title={{SPARTUN}3D: Situated Spatial Understanding of 3D World in Large Language Model},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=FGMkSL8NR0},
note={under review}
}
Integrating the 3D world into large language models (3D-based LLMs) has been a promising research direction for 3D scene understanding. However, current 3D-based LLMs fall short in situated understanding due to two key limitations: 1) existing 3D datasets are constructed from a global perspective of the 3D scenes and lack situated context.
2) the architectures of the current 3D-based LLMs lack an explicit mechanism for aligning situated spatial information between 3D representations and natural language, limiting their performance in tasks requiring precise spatial reasoning. 
In this work, we address these issues by introducing a scalable situated 3D dataset, named Spartun3D, that incorporates various situated spatial information.
In addition, we propose a situated spatial alignment module to enhance the learning between 3D visual representations and their corresponding textual descriptions. Our experimental results demonstrate that both our dataset and alignment module enhance situated spatial understanding ability.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Situated Understanding in 3D Scen 3D VL LLM
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/cd742aa6d024250fa72188e7f6368f931a792a72.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
SPARTUN3D: Situated Spatial Understanding of 3D World in Large Language Model
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission