MAPS: Advancing Multi-Modal Reasoning in Expert-Level Physical Science
multi-modal reasoning;scientific reasoning;physical simulation
Yes
No ethics review needed.
Incorrect spelling and expression:
- Line 243 - diagram diagram -> diagram
- Appendix A.2, Figure 5, Row2 - Chnese -> Chinese

Some Confusions:
The research idea proposed in this paper is highly innovative, but during my reading, I encountered several questions that I hope the authors can provide answers to:
- Table 1 presents the conversion effectiveness of PPM on the synthetic dataset ppm-syn-lprc-test and the real-world dataset SimpleCircuitEval. It is noted that the recognition accuracy of Label-Type diagrams is lower than that of Numerical-Type diagrams, which seems contrary to common belief since label-type diagrams are generally considered easier to identify than numerical-type diagrams. What is the reason for this difference?
- The test dataset SimpleCircuitEval comprises only 79 questions. Is this sample size sufficient to support a convincing evaluation result? Does it adequately represent the universality of LPRC question types?
The advantages of this paper are mainly reflected in the following aspects:
- This paper proposes an innovative process framework that can combine physical perception models (PPMs) with simulator outcomes to infer answers to physical problems. The framework integrates the understanding of physical diagrams with the reasoning of physical knowledge, and its effectiveness has been validated through experiments;
- The paper designs and introduces a synthetic dataset named ppm-syn-lprc, which is used for fine-tuning the visual-language model PPM. The generation process of the synthetic dataset is elaborated in detail, and the scalability of this method is demonstrated. For instance, this approach can be applied to other fields such as mechanical systems, providing clear directions and ideas for improving the model's reasoning capabilities in other areas of physics;
- The structure of the paper is well-organized and logical, making it easy for readers to understand and follow.
This paper proposes an innovative framework named MAPS, which relies on multimodal large language models (MLLMs). During the inference phase, it integrates the simulated language descriptions of input diagrams generated by PPM and data obtained through the Chain-of-Simulation process, aiming to address complex structural and quantitative analysis issues in the field of physics. Through empirical studies on university-level circuit analysis problems, the authors demonstrate the significant effect of MAPS in enhancing the inference accuracy of MLLMs, outperforming all existing models. These results reveal the potential of MAPS in strengthening the multimodal scientific reasoning capabilities of MLLMs, providing a promising direction for the development of this field.
Writing:
- There are some typos in the article, and it is recommended that the author carefully proofread and corrected them to enhance the professionalism and readability of the paper.

Experimental Design:
- Generalization Issues: The results of this study have only been validated on the GPT-4V model, which may not be sufficient to demonstrate the applicability of the framework to other model architectures. It is suggested that the authors extend the evaluation of the framework to different model architectures to strengthen the generalizability and credibility of the experimental conclusions.
- Evaluation of Synthetic Data Effectiveness: The paper mentions using CogVLM-17B as the base model for PPM and fine-tuning it on the synthetic dataset. To clarify whether the improvement in the model’s ability to understand physical diagrams is due to the inherent capabilities of the base model or the introduction of synthetic data, the authors are advised to add comparative experiments before and after fine-tuning. This will help verify the effectiveness of synthetic data and reinforce the persuasiveness of the research findings.
Yes
No ethics review needed.
See the weakness section.

The prompts in the Appendix mention "Chinese circuit problem".  What is meant by "Chinese circuit problem"? Is it a specific term? If not, perhaps you can revise its usage to avoid potential confusion.
The paper makes several contributions. It creates a large dataset of diverse circuit diagrams. 
It proposes to decompose expert-level multi-modal reasoning task into physical diagram understanding via a Physical Perception Model (PPM) and reasoning with physical knowledge via a simulator. (This is a good approach.)
The paper show that MAPS improves reasoning accuracy in the electronic circuit analysis domain. 
The authors point out that all the information about a circuit may not be in the diagram and may be in the surrounding text and their approach addresses this issue.
This paper presents the framework of Multi-Modal Scientific ReAsoning with Physics Perception and Simulation (MAPS) based on an MLLM. It tries to decompose expert-level multi-modal reasoning task into physical diagram understanding via a Physical Perception Model (PPM) and reasoning with physical knowledge via a simulator.  The authors show that MAPS improves reasoning accuracy in the electronic circuit analysis domain. Towards that end, the authors craft rules to generate a large dataset of diverse circuit diagrams and their corresponding simulation language descriptions. This synthetic data is then used to fine-tune a pre-trained VLM, ultimately producing the PPM.
It is not clear where errors happen. An error analysis of a sample of the evaluation data would be helpful. In particular,  please provide breakdown of errors by type (e.g., perception errors vs. reasoning errors), and give examples of common failure cases.

What are the different kinds of questions addressed in the paper? Do they need simulation or just a solver. (With simulation capability many more complex questions can be answered using multiple simulations.) Please provide a categorization of the question types in your dataset, along with examples of each category. In your current dataset what percentage of questions requires a single simulation, multiple simulations, and ones that can be solved with simpler methods.

What kind of training data is used with respect to the questions and Simulation language specification? More information on that would be helpful. In particular, please provide information such as the distribution of question types, and the complexity of circuits represented.
Yes
No ethics review needed.
The paper demonstrates the effectiveness of MAPS using circuit analysis problems. Can the authors provide evidence or insights into how MAPS could be adapted for other physical sciences, such as mechanics or optics, which involve different types of diagrams and simulation languages?
S1: The integration of perception and simulation for multi-modal scientific reasoning is well-conceived and leverages MLLM strengths while mitigating their weaknesses in handling complex diagrams.

S2: Results from circuit analysis problems highlight a notable increase in accuracy, showcasing MAPS' ability to outperform current state-of-the-art methods.

S3: The paper provides comprehensive explanations for data synthesis, PPM training, and the inference process, enhancing reproducibility.
This paper presents MAPS (Multi-Modal Scientific Reasoning with Physics Perception and Simulation), a novel framework designed to improve the performance of Multi-Modal Large Language Models (MLLMs) in expert-level scientific reasoning tasks, specifically in physical sciences. MAPS enhances the comprehension and analytical processes by integrating a Physics Perception Model (PPM) with a simulator for interpreting complex physical diagrams and quantitative reasoning.
W1: The framework is tested primarily on circuit analysis, which may not fully capture its adaptability across different physical sciences.

W2: The multi-step process involving diagram conversion, SL generation, and simulation may introduce cumulative errors, which could affect real-world applicability.

W3: The reliance on synthetic data poses a challenge for real-world accuracy, as unseen or complex diagrams might not align with generated examples.
Yes
No ethics review needed.
None.
1. There is some novelty in augmenting MLLM with the formal languages describing the circuits and using an external physics simulator.
2. The proposed method works well in improving the performance.
3. The paper is written clearly and easy to follow
The paper proposes to use improve Multi-Modal LLM's reasoning capability by fine-tuning with domain-specific formal languages and using external physics simulator.
1. The paper only proposed a solution for a very specific domain of physical reasoning, namely circuit analysis. This implies that for every other sub-domain, we need to use another set of formal language and physics simulator, which is not scalable. This also implies that this philosophy cannot be extended to physical reasoning problems that don't have any formal languages describing it, which reduces its potential in the frontiers of science research.
improving multi-modal scientific reasoning capability with physics perception model and simulation assistance
@inproceedings{
anonymous2024maps,
title={{MAPS}: Advancing Multi-Modal Reasoning in Expert-Level Physical Science},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=GR0y0F3Ipd},
note={under review}
}
Pre-trained on extensive text and image corpora, current Multi-Modal Large Language Models (MLLM) have shown strong capabilities in general visual reasoning tasks. 
However, their performance is still lacking in physical domains that require understanding diagrams with complex physical structures and quantitative analysis based on multi-modal information. 
To address this, we develop a new framework, named **M**ulti-Modal Scientific Re**A**soning with **P**hysics Perception and **S**imulation (**MAPS**) based on an MLLM. 
MAPS decomposes expert-level multi-modal reasoning task into physical diagram understanding via a Physical Perception Model (PPM) and reasoning with physical knowledge via a simulator. 
The PPM module is obtained by fine-tuning a visual language model using carefully designed synthetic data with paired physical diagrams and corresponding simulation language descriptions. 
At the inference stage, MAPS integrates the simulation language description of the input diagram provided by PPM and results obtained through a Chain-of-Simulation process with MLLM to derive the underlying rationale and the final answer. 
Validated using our collected college-level circuit analysis problems, MAPS significantly improves reasoning accuracy of MLLM and outperforms all existing models. 
The results confirm MAPS offers a promising direction for enhancing multi-modal scientific reasoning ability of MLLMs. 
We will release our code, model and dataset used for our experiments upon publishing of this paper.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
multi-modal reasoning scientific reasoning physical simulation
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/b476721fd383b5aa078a4b88fc17fc604a89fad4.pdf
applications to physical sciences (physics, chemistry, biology, etc.)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
MAPS: Advancing Multi-Modal Reasoning in Expert-Level Physical Science
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission