FoREST: Frame of Reference Evaluation in Spatial Reasoning Tasks
Spatial language;Evaluation benchmark;Frame of reference
Yes
No ethics review needed.
Does the temperature setting impact the bias of LLMs on this task? The paper sets the sampling temperature to 0, and claims the bias of LLMs by showing that they more frequently produce external classes under ambiguous queries (that correspond to multiple correct categories) in Figure 4. It is possible that a low temperature limits the diversity of LLMs response.
The paper proposes a novel task that can potentially reveal the ability of LLMs in understanding spatial concepts.
This paper proposes a new Frame of Reference (FoR) comprehension task for LLMs, where the models need to identify the perspective category based on a given spatial narrative. For this task, a new benchmark, named FoREST is generated. Using this benchmark, the paper identifies the inability and biases of various LLMs in solving this task, and proposes a new prompting technique to guide LLMs in identifying key information from the textual input thus enhancing their performance on this task. This paper also shows how this ability can be utilized for text-to-image generation under specific spatial guidance, highlighting the potential application value of this work.
1. It is unclear if the proposed Spatial-Guided prompting technique helps “reduce FoR bias in LLMs”, as claimed in the abstract in the paper, or just clarify the category terms that LLMs are tasked to identify. Since the FoR classes (external intrinsic, external relative, etc) are technical terms in cognitive studies that do not appear commonly in the internet data used for training LLMs, a clear and intuitive explanation of the terms is naturally important for solving this task. However the definition of the terms provided to the LLMs is formal and not intuitive. For example, it does not define clearly what does “the referenced object’s intrinsic directions” mean. This is only explained to some extent in the Spatial-Guided prompting examples, such as “the car has direction”. What if the concepts are explained in plainer and more intuitive language? Such as:

$\textit{"External intrinsic: The spatial description of an item A relative to another item B, where (1) A is not contained by B; (2) The spatial relationship description is relative to the B’s facing direction, if B has one (Example: a horse, a car. Counterexample: a box.)"}$

2. Why understanding FoR is an important problem is not articulated adequately. Since, according to the introduction in this paper, this task is more commonly seen cognitive linguistics study rather than AI or related fields, more discussion on the potential application of FoR understanding ability of LLMs can help readers better understand the motivation. The text-to-image task shown in the paper is a great application, but it is on specifically designed command types. Can this ability be potentially applied for other embodied AI or robotic tasks that require strong spatial understanding capacity?
Yes
No ethics review needed.
Could you make the datasets and code available?
- **Novel Perspective**: Introduces an innovative approach to assessing spatial perception in large models, focusing on frames of reference (FoR).
- **Theoretical Support**: Draws on established spatial language literature to support the motivations and foundational concepts of FoR.
- **Insightful Analysis**: Offers valuable insights into both FoR identification and text-to-image mapping.
The paper presents the FoREST benchmark, aimed at testing large language models' (LLMs) understanding of frames of reference (FoR) in spatial reasoning. FoR refers to different perspectives (intrinsic, relative, and absolute) used to describe spatial relationships. The benchmark assesses LLMs' ability to identify FoR in ambiguous and clear spatial contexts and perform text-to-image generation accordingly. Results show that LLMs have biases in FoR interpretation, impacting spatial layout accuracy. They also introduce Spatial-Guided prompting to improve FoR comprehension and performance in related tasks.
No dataset or code provided
Yes
No ethics review needed.
1. Is SG-prompting zero-shot or few-shot? In T2I task, what are examples mentioned in line 290?
2. In table 4, it's abnormal to see GPT-3.5 outperforms GPT-4o in A-split. And why do GPT family models do exceptionally well in EI and II C-split, yet relatively bad at ER and IR C-split? It would be interesting to dive into these observations.
1. Spatial ability of LLMs are an important research topic yet less explored. A scientific benchmark would contribute to this area.
2. This work conducts various experiments with a range of LLM models and provide in-depth analysis. It also verifies the proposed prompting method in text-to-image task, adding its value to real world applications.
3. The paper is well organized and written.
This paper introduces a new benchmark called FoREST to evaluate LLMs spatial ability in understanding "Frame of Reference". Additionally, it proposes Spatial-Guided prompting to improve the FoR classification task and layout generation in text-to-image generation.
1. The dataset is pure synthetic and constructed by a limited number of textual templates. I have concerns about the FoR classification task given the template "<locatum> <spatial relation> <relatum> <perspective>". It seems hard to disentangle this task with linguistic and common-sense reasoning of LLMs. For example,  LLMs are able to determine whether the perspective is intrinsic or relative by analyzing perspective template, and analyze topology template to determine whether the locatum is external or internal. Both of them don't necessitate understanding the underlying spatial configuration under a specific perspective. On the contrary, the text-to-image task indeed requires the model to interprete the spatial configuration and transform the perspective to camera's.
2. Again, since the dataset is synthetic and constructed by textual templates, the inductive bias might be leveraged by SG-prompting. 
3. Lack of full prompts of different settings, such as few-shot, CoT, SG-prompting, text-to-layout and SG to layout.
Yes
No ethics review needed.
I would look forward to seeing more experimental results, particularly on how different large language models perform on the benchmark. For example, how do the latest models like Qwen2 or Molmo perform in different cases? Additionally, it’s intriguing that spatial understanding may decline as parameter count increases—what could be the underlying reasons? I also noticed that the performance decreases with the use of CoT and 4-shot settings, which is puzzling. What might be causing this effect?
This paper focuses on a core shortcoming in large language models' text comprehension-spatial understanding, presenting a benchmark for evaluating spatial perception abilities. It summarizes potential situations in spatial perception using existing evaluation metrics. Downstream tasks incorporate text-to-image generation experiments with diffusion models to visualize the difference in spatial understanding capabilities of different LLMs. The paper is well-structured, with straightforward explanations of the methods, and employs vivid examples to enhance understanding.
Research on spatial perception capabilities in large language models is a key direction for optimizing their generation abilities. This paper introduces a benchmark for understanding frames of reference (FoR) and evaluates different LLMs to test the spatial perception capabilities. Additionally, it uses diffusion models to conduct experimental visualizations that simulate this understanding. The benchmark provides guidance on designing prompts that enhance spatial reasoning, contributing to improved text-to-image accuracy.
However, this paper lacks logical coherence in its descriptions of various cases. The selection of content for the four FoR classes seems random, raising the question of whether there could be a clearer division for categorizing different spatial reasoning tasks. It remains unclear if the four cases can comprehensively cover all possible spatial reasoning tasks, and more citations are needed to support your claim. The presentation of experimental results is also insufficient; for instance, the two images in Figure 3 fail to follow the principle of controlling variables, rendering the conclusions unconvincing. Furthermore, the experimental analysis is inadequate; while the comparison between LLaMA3-8B and LLaMA3-70B results is noteworthy, the spatial understanding of LLaMA declines as parameters increase, varying across C-Split cases. This raises the question of what insights researchers can draw from this benchmark to adjust datasets to maintain or even improve spatial understanding performance. Addressing this issue is essential to the benchmark's purpose.
@inproceedings{
anonymous2024forest,
title={Fo{REST}: Frame of Reference Evaluation in Spatial Reasoning Tasks},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=9Y6QWwQhF3},
note={under review}
}
Spatial cognition is one fundamental aspect of human intelligence. A key factor in spatial cognition is understanding the frame of reference (FoR) that identifies the perspective of spatial relations. 
However, the AI research has paid very little attention to this concept.
Specifically, there is a lack of dedicated benchmarks and in-depth experiments analyzing large language models' (LLMs) understanding of FoR.
To address this issue, we introduce a new benchmark, **F**rame **o**f **R**eference **E**valuation in **S**patial Reasoning **T**asks (FoREST)  to evaluate LLMs ability in understanding FoR.
We evaluate the LLMs in identifying the FoR based on textual context and employ this concept in text-to-image generation. 
Our results reveal notable differences and biases in the FoR identification of various LLMs. 
Moreover, the bias in FoR interpretations impacts the LLMs' ability to generate layouts for text-to-image generation. 
To deal with these biases, we propose Spatial-Guided prompting, which guides the model in exploiting the types of spatial relations for a more accurate FoR identification. 
This approach reduces FoR bias in LLMs and improves the overall performance of FoR identification. 
Eventually, using FoR information in text-to-image generation leads to a more accurate visualization of the spatial configuration of objects.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Spatial language Evaluation benchmark Frame of reference
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/5da4e30a54fba17c1b2a0568defbac096401402d.pdf
datasets and benchmarks
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/98624b5c45b096fbaa88ebd541aeb1a3e8d2c5fd.zip
FoREST: Frame of Reference Evaluation in Spatial Reasoning Tasks
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission