R-CoT: Reverse Chain-of-Thought Problem Generation for Geometric Reasoning in Large Multimodal Models
Large Multimodal Models;Mathematical Reasoning;Geometric Reasoning
Yes
No ethics review needed.
Please refer to Weaknesses
1. The multimodal mathematics direction is challenging, and dataset construction for geometric problems is meaningful. 
2. The paper is clearly written and easy to follow.
The authors present a two-stage Reverse Chain-of-Thought (R-CoT) pipeline to enhance mathematical geometric reasoning in large multimodal models (LMMs). First, they introduce GeoChain, which generates geometric images and descriptions that highlight relationships between geometric elements. This is followed by a Reverse A&Q method, which uses step-by-step reasoning based on these descriptions to generate questions in reverse from the reasoning outcomes.
1. Overclaim: The statement, "Notably, R-CoT-8B significantly outperforms previous state-of-the-art open-source mathematical models by 16.6% on MathVista and 9.2% on GeoQA, while also surpassing the closed-source model GPT-4o by an average of 13% across both datasets," appears overclaim. Given that this work primarily focuses on data generation, fair assessment would require consistent model comparison settings. The claim in the abstract is misleading due to differences in model size and backbone. For instance, the authors use an 8B model as the backbone, whereas prior methods rely on a 7B model. As shown in Table 1, improvements under comparable settings are marginal.
2. Concern of Entire Data Quality: In Table 1, combining your data with Geo170K yields only marginal improvements, which raises questions about the efficacy of the GeoMM dataset. Can the authors provide results that reflect the performance of using only GeoMM?
3. Image Diversity: Clarification is needed on how images generated by GeoChain differ from those in existing datasets. For example, the images shown in Figure 12 (Examples of GeoMM dataset) appear similar to datasets like GeoQA and Geo3K. The marginal improvement observed further suggests potential alignment or overlap with existing data domains, calling into question the added diversity. Moreover, how is diversity measured?
4. Concern of QA Quality: The synthetic questions are annotated using ERNIE Bot 4.0. After generating the QA pairs, what steps were taken to ensure their correctness, and has this been verified?
5. Question about Method: When segmenting descriptions into patches for reasoning, it can be challenging to assess the difficulty level of each condition in relation to others. For any given description, how is the hierarchy of condition difficulty determined? Additionally, how do you ensure that each subsequent condition is logically derived from previous ones to yield varied difficulty levels in the answers?
6. Writing: In Table 4, it’s noted that the baseline models vary across R-CoT models of different sizes, yet comparisons mention only the model sizes, which creates an unfair comparison. Stating specific settings for both your models and baseline models (LLMs/VLMs) in the table is necessary to give readers an accurate understanding of performance.
Yes
N/A
No ethics review needed.
1.	In Section 3.1, the paper states that GeoChain generates "high-fidelity" images. What criteria or metrics were used to define and verify this fidelity?
2.	The description generation for the images lacks detailed criteria for ensuring the accuracy. What specific metrics or validation methods were employed to assess the fidelity of these descriptions? In cases where the generated image descriptions were inaccurate or lacked detail, how were these issues addressed?
3.	How does line sampling affect image quality and the diversity of generated images? Could the authors explain the effect of varying the number of sampling rounds?
4.	Could the authors provide an analysis of common error types in generated Q&A pairs and any steps taken to minimize them?
5.	In Section 3.2, can the authors clarify how “single-step reasoning results” are progressively fused in Chain-of-Thought Fusion? Could examples or intermediate reasoning steps be provided to clarify this fusion process?
6.	Given the two-stage process of GeoChain and Reverse A&Q, can the authors provide an analysis of the computational cost and time efficiency of R-CoT? 
7.	Are there any specific geometric problem types where R-CoT struggles?
1.	The R-CoT pipeline introduces a structured way to generate geometric problems by reversing the usual chain-of-thought, which enhances data accuracy and diversity. This approach aligns well with complex reasoning tasks in geometric settings.
2.	R-CoT achieves improvements in the 2B, 7B, and 8B model settings, outperforming existing open-source and closed-source models, such as GPT-4o, particularly on benchmarks like MathVista and GeoQA.
The paper proposes Reverse Chain-of-Thought (R-CoT), a method to improve geometric reasoning in Large Multimodal Models by generating high-quality image-text data through a two-stage process. First, the GeoChain stage creates precise geometric images with descriptive relationships between elements. Then, the Reverse Answer & Question stage generates questions in reverse, based on step-by-step reasoning from these descriptions. The results show improved model performance on benchmarks like MathVista and GeoQA.
1.	The reverse chain-of-thought concept is essentially a reconfiguration of existing Chain-of-Thought (CoT) techniques applied to geometric data generation. The novelty is thus limited, as R-CoT primarily adapts and combines established ideas without introducing a fundamentally new approach. 
2.	The effectiveness of R-CoT heavily relies on the fidelity of the generated images and descriptions. However, the process for validating the quality and accuracy of these generated descriptions seems insufficiently detailed, especially given the complexity of geometric reasoning.
3.	The R-CoT pipeline involves a multi-stage generation and reasoning process, which could increase computational cost. It is unclear whether this approach offers efficiency or if the computational demands outweigh its benefits.
Yes
No ethics review needed.
1. Can you provide more analysis on the role of the vision part generated by GeoChain in solving the geometric problems?
1. The R-CoT approach is inventive in using a reverse process for generating questions.
2. The GeoChain component improves upon traditional synthetic data generation methods by increasing visual realism and complexity in the generated geometric problems.
The paper introduces R-CoT, a two-stage geometry problem generation pipeline aimed at enhancing geometric reasoning in large multimodal models. The approach involves generating high-fidelity geometric images and associated descriptions using GeoChain and subsequently creating questions through a reverse process using an LLM-based Reverse A&Q method. The authors claim substantial performance gains on MathVista and GeoQA benchmarks, with R-CoT-8B surpassing state-of-the-art performance on open-source and even closed-source baselines.
1.	**Unclear Necessity of the Vision Modality**:
The integration of a vision component within this framework raises concerns regarding its necessity and effectiveness. The core of the geometric problem-solving process, as presented, appears to rely on CoT reasoning based solely on textual descriptions. Given that the problems are fully described in the text, the role of the generated geometric images in augmenting reasoning is ambiguous. Specifically, the added visual modality does not clearly offer unique insights or support that could not otherwise be inferred through text alone. Without a compelling demonstration of how the visual component uniquely facilitates problem-solving, its inclusion seems unnecessary and potentially redundant.

2.	**Ambiguity in Difficulty Level Classification**:
The criteria used for categorizing the problems into three difficulty levels—easy, medium, and hard—are insufficiently defined and appear somewhat arbitrary. For instance:

- *Inconsistent Alignment with Difficulty Criteria*: The difficulty classification seems to focus on the number of reasoning steps, but without concrete justifications, this metric alone may not adequately capture the true complexity. The nature of geometric reasoning can vary widely, and simply having more reasoning steps does not necessarily equate to a harder problem. 

- *Spatial Reasoning as a Key Factor*: For certain problems that require interpreting spatial relationships or auxiliary constructs (e.g., adding segments for further reasoning), a visual component might be essential. These types of problems could then reasonably fall under higher difficulty levels if they require the vision modality to infer spatial arrangements. However, the paper does not fully clarify when and why visual input would be required in these cases.

Additionally, according to [Ref. 1], those data correctly generated by CoT of LLMs can be considered within a Completely Feasible Reasoning Boundary (CFRB), which characterizes the simple reasoning tasks. Therefore, the difficulty criteria should be re-evaluated to better suit the geometry task, accounting for situations where visual input may or may not be essential.

[Ref. 1] Unlocking the Capabilities of Thought: A Reasoning Boundary Framework to Quantify and Optimize Chain-of-Thought. NeurIPS 2024.
We propose a two-stage Reverse Chain-of-Thought (R-CoT) geometry problem generation pipeline to synthesize high-quality mathematical geometry data.
@inproceedings{
anonymous2024rcot,
title={R-CoT: Reverse Chain-of-Thought Problem Generation for Geometric Reasoning in Large Multimodal Models},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=iwVkB9zaVb},
note={under review}
}
Existing Large Multimodal Models (LMMs) struggle with mathematical geometric reasoning due to a lack of high-quality image-text paired data. Current geometric data generation approaches, which apply preset templates to generate geometric data or use Large Language Models (LLMs) to rephrase questions and answers (Q&A), unavoidably limit data accuracy and diversity. To synthesize higher-quality data, we propose a two-stage Reverse Chain-of-Thought (R-CoT) geometry problem generation pipeline. First, we introduce GeoChain to produce high-fidelity geometric images and corresponding descriptions highlighting relations among geometric elements. We then design a Reverse A&Q method that reasons step-by-step based on the descriptions and generates questions in reverse from the reasoning results. Experiments demonstrate that the proposed method brings significant and consistent improvements on multiple LMM baselines, achieving new performance records in the 2B, 7B, and 8B settings. Notably, R-CoT-8B significantly outperforms previous state-of-the-art open-source mathematical models by 16.6% on MathVista and 9.2% on GeoQA, while also surpassing the closed-source model GPT-4o by an average of 13% across both datasets.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Large Multimodal Models Mathematical Reasoning Geometric Reasoning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/b111d64db48f13067d5722f320d1e93310c1dfad.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
R-CoT: Reverse Chain-of-Thought Problem Generation for Geometric Reasoning in Large Multimodal Models
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission