Visual Agents as Fast and Slow Thinkers
Multimodal Large Language Model;System2 Thinking;Language Agent
Yes
No ethics review needed.
Please see weaknesses above. I believe the paper in its current form already presents promising ideas+results but can be made still stronger.

– It would be good include a discussion / conceptual comparison to recent “large reasoning models” that propose scaling test-time compute eg. the OpenAI o1 model. While not (yet) multimodal, such methods are speculated to be trained w/ reinforcement learning and to produce “internal” (i.e. not necessarily interpretable) chain of thought – what are some of the tradeoffs of this type of approach against FaST?

– [Minor] As I understand, FaST does not provide increased transparency when it opts for “fast” thinking – it would be good to clearly demarcate this.

– [Minor] L538 typo: extra "A"
– The paper presents a highly novel solution to an important problem

– The paper is very well-written and easy to follow

– The motivation for the approach is laid out very clearly and is compelling

– The proposed approach  is designed intuitively, and shows convincing improvements over prior work

– The neuro-symbolic nature of the approach directly translates to improved transparency

– The paper presents a comprehensive set of results and ablation studies on visual question answering and referring and reasoning segmentation benchmarks. The qualitative results included in the paper are also very compelling

– The paper includes a detailed appendix and code implementation which will aid in reproducibility
Proposes FaST, a Fast-and-Slow-Thinking mechanism for vision-language models that models both “system 1” (fast) and “system 2” (slow) thinking, dynamically switching between the two based on query complexity. The method demonstrates improvements over baselines on standard benchmarks in addition to improved reasoning and transparency.
– [Complexity] One concern I have is the complexity of the approach, especially considering the large number of moving parts. Do the same set of method hyperparameters generalize across settings, or was a large amount of tuning required to achieve the results reported in the paper? 

– [Clarity] I found some of the method details to be lacking eg. L259-260 mentions “a fast sampler based on cross-attention” but I could not find any further details for this component.

– [Performance] While FaST does seem to consistently outperform the LLaVA-1.5 base approach that it builds on top of, it is not exactly an apples-to-apples comparison considering the additional models /  datasets that are used (including large ones such as SAM). Meanwhile, its gains over methods reasoning-focused approaches like Visual CoT and Chain-of-Spot are modest (and sometimes lags considerably eg. behind VCoT on VQAT) – a deeper analysis of / comparison against these methods (eg. average runtime etc.) would be helpful to establish the pros/cons of using FaST. The comparison against CoVLM and CogCOM appears more significant (Fig 4) but is missing a comparison to the LLaVA-1.5 baseline w/ the stronger encoder. 

– [Analysis] The results presented in Figure 5 are very interesting, but are missing baseline performance of the base LLaVA-1.5 model: what is its accuracy on the subset of system 2 / system 1 questions (as judged by FaST) in GQA/MME? How much does “slow” thinking actually improve performance on such questions (assuming that the switch adapter is accurate – it would also be good to validate this eg. by collecting question complexity labels)?

– [Analysis] It would be valuable to include an evaluation of the correctness/quality of the “chain-of-evidence” generated by the approach, perhaps via a human study and/or model interpretability methods. Does the chain-of-evidence appear faithful or is it mostly a post-hoc rationale? How often does a “correct” chain of evidence lead to the right conclusion, and vice versa? Is the chain-of-evidence used for the model “optimal”, or does it include unnecessary steps or logical leaps? This will help build confidence around the improved transparency claims made in the paper.
Yes
No ethics review needed.
1. Can the authors provide more details in the synthesis of the dataset used for switch adapter finetuning? Specifically, how are the negative samples from V* augmented to cover the two cases that are expected to trigger a slow thinking mode switching (uncertainty in pinpointing objects or the targets are too small)?    
2. Will the datasets collected by the authors be released? The reviewer believes that these datasets are crucial for reproducing the FAST framework proposed in the paper, and they will also be highly beneficial for future related work.
3. Can the authors provide some experimental results for other reasoning tasks, like raven, bongard or physical reasoning tasks?
1. FaST strikes a balance between accuracy and efficiency for visual agents by dynamically integrating fast and slow reasoning. The motivation to incorporate different thinking modes based on the scenario is intuitive, and the primary concept is presented clearly and logically.
2. The interpretability and transparency of the proposed framework are improved through the introduction of a chain-of-evidence mechanism and the generation of symbolic intermediate results as feedback.
3. Experimental results across various tasks validate the effectiveness of the proposed framework.
The paper presents FaST, a framework for improving the cognitive capacities of visual agents.   
Inspired by System 1 and System 2 thinking in human’s cognition, FaST incorporates a switch adapter that dynamically toggle between a faster, intuitive response mode and a slower, analytical reasoning mode depending on the task complexity. 
FaST also features a high interpretability by integrating hierarchical reasoning strategies and symbolic intermediate outputs into the framework.
Experiments show that the proposed outperforms baselines in various VQA and multimodal benchmarks.
1. The proposed framework primarily focuses on VQA and segmentation tasks. To better mirror System 2 recognition, the reviewer suggests evaluating results from more complex reasoning tasks (e.g., the Bongard problem, physical reasoning tasks) to further validate the generalization capabilities of the proposed method.
2. The efficacy of the switch adapter can be further analyzed in Section 3.2:
  * No investigations have been conducted on the switching ratio and performance on reasoning segmentation and other tasks, which is pivotal to demonstrate the universal compatibility of the switch adapter mechanism across different families of tasks.
  * The description of accuracy rates “under System 2 mode” contains some ambiguities. The authors should clarify in the text that accuracy rates reported under System 2 are also evaluated using fast thinking mode, otherwise, the claim regarding the adapter’s ability to differentiate problem complexities is not adequately supported. 
  * The accuracy rates mentioned in the passage do not appear to match those shown in Figure 5.
Yes
No ethics review needed.
The major concerns are listed in the weakness part.

Minimal concern: It is definitely not a good idea to use "fast" as an acronym for "fast" and "slow".

Potential Typos:
- Lines 151-156: Format like "Equation 1" does not align with Sec.2.2 "Eq.3"
- Line 160: "Def.2.2", only Remark 2.2 exists
- Line 196: "Eq.2.1", only Remark 2.1 exists
- Line 215: "Eq.2.4", only Remark 2.4 exists
- The method is built based on the idea of Kahneman's system 1/2 theory from cognitive science, which is clearly-motivated and reasonable.

- The approach is clearly written and its implementation is reasonable.

- The authors provide comprehensive evaluation across multiple benchmarks and tasks. And the method outperforms baselines.

- The authors provide clear visualizations of the model performance.
The paper proposes a slow-fast paradigm for MLLMs based on Kahneman's System 1/2 therory. The framework consists of a "fast" module built upon MLLM and vision foundation models. A switch adaptor will determine if it's necessary to use "slow" module, which will gather all detailed clues to find a more accurate result. The approach shows impressive results across multiple benchmarks and outperforms baselines models.
- The analogy between human cognitive system 1-2 and the "fast" "slow" module in the proposed method lacks rigor justification, or even misaligned. In human cognition, System 1 usually handles low-level tasks and is much faster, while System 2 is slow but handles complexity. However, in FaST, both the fast and slow modules solves same problem, but only have a tradeoff between efficiency and performance. Considering that the "slow" module is only 4x slower, we can abandon the switch adaptor and use the slow module all the time for a maximal performance.
- From another perspective, if the only purpose of two stage model is the computation efficiency, then the proposed approach is more like an early-exit trick of a complex model (referring to the "slow" module). THis diminish the novelty of the paper.
- The framework essentially implements a manual Chain of Thought (CoT) approach rather than dual cognitive system. If reframing the method in to CoT paradigm, a progressive visual cue integration would be potentially developed (more than 2 layers). This approach would gradually add visual evidence into the reasoning process, which could be both more efficient and accurate.
- The complexity burden is only **shifted** to the switch adaptor, rather than being alleviated or addressed. So what is the performance of the switch adaptor itself?
- The details of the datasetin remark 2.1 is missing. It is critical to introduce the construction process of the dataset.
Yes
No ethics review needed.
Overall, I found the paper very interesting, my main concern is within the weak points. Another question I have is, if this FaST can outperform other models by using only System 2, what are the advantages of switching between System 1 and System 2 instead of just System 2?  I think time cost would be an answer, but I'd be curious if the author provides more answers.




typo: In Line 538, AFurthermore should be Furthermore
1. The motivation for incorporating systems 1 and 2 in visual agents seems promising and convincing. Especially the design of system 2, the introduction of clue, seg instance and chain of evidence can assist the reasoning process based on experiment results. The construction of Negative Data for Target Objects Reasoning Dataset seems useful for training the adapter switcher. 
2. The hierarchical reasoning capabilities that start with global image cues and drill down to pixel-level analysis provide a well-rounded method of handling visual inputs. The symbolic approach can provide more explainability on the model reasoning process.
2. A comprehensive experiment, including many current models, proves the effectiveness of FaST. The ablation study on pipeline structure and the time comparison highlight the advantage of FaST.
3. Overall, I found the paper well written, with clear definitions and formulas to demonstrate the complex system. The motivation based on human cognitive science is well presented.
The paper introduces FaST, a framework based on human reasoning that incorporates fast (system 1) and slow (system 2) thinking mechanisms into visual agents. FaST involves a switch adapter to change between systems 1 and 2 based on the complexity of the query question. While System 1 reasons the question in a short or one-step approach, system 2 uses a switch adapter to provide clues and object names, which later inspires the proposal adapter to provide region boxes and seg adapter to provide seg instances. In the end, all additional information will be part of a chain of evidence and prompt LLMs to do "slow" reasoning to output an answer. The paper does a comprehensive experiment comparing FaST with current models and demonstrates the effectiveness of FaST on both visual question answering and segmentation reasoning. Further ablation studies prove the significance of each component within the Fast.
1. Result Demonstration: based on the paper, I think we can assume the previous model mainly uses fast thinking on reasoning questions. A better way to illustrate the effectiveness of FaST is to split Table 1 with the model's system 1 and system 2 performance. In other words, for all those questions in which FaST use system 1 reasoning, compare the performance in one table and compare all questions with system 2 in another table.  I expect the improvement of FaST mainly comes from system 2, which means FaST will show similar performance with other models in questions that require system 1 reasoning.
2. Dependence on Specific Datasets: I appreciate the work constructing the “Negative Data for Target Objects Reasoning Dataset" to train the switch adapter. However, I am concerning the training data can not reflect all real-world scenario, which may limit generalizability. Is it possible for us to gain the signal from the model's inner representation to decide the switch between systems 1 and 2?
3. Failure case: the paper should also explore scenarios where it fails or underperforms compared to baseline models. As the FaST use a hierarchical reasoning pipeline, it is possible the error of some component can make the whole system collapse or lead to a cascade of issues. An analysis of a failure case can provide insight into the model's weaknesses and future directions.
@inproceedings{
anonymous2024visual,
title={Visual Agents as Fast and Slow Thinkers},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=ncCuiD3KJQ},
note={under review}
}
Achieving human-level intelligence requires refining cognitive distinctions between \textit{System 1} and \textit{System 2} thinking. While contemporary AI, driven by large language models, demonstrates human-like traits, it falls short of genuine cognition. Transitioning from structured benchmarks to real-world scenarios presents challenges for visual agents, often leading to inaccurate and overly confident responses. To address the challenge, we introduce \textbf{\textsc{FaST}}, which incorporates the \textbf{Fa}st and \textbf{S}low \textbf{T}hinking mechanism into visual agents. \textsc{FaST} employs a switch adapter to dynamically select between \textit{System 1/2} modes, tailoring the problem-solving approach to different task complexity. It tackles uncertain and unseen objects by adjusting model confidence and integrating new contextual data. With this novel design, we advocate a \textit{flexible system}, \textit{hierarchical reasoning} capabilities, and a \textit{transparent decision-making} pipeline, all of which contribute to its ability to emulate human-like cognitive processes in visual intelligence. Empirical results demonstrate that \textsc{FaST} outperforms various well-known baselines, achieving 80.8\% accuracy over $VQA^{v2}$ for visual question answering and 48.7\% $GIoU$ score over ReasonSeg for reasoning segmentation, demonstrate \textsc{FaST}'s superior performance. Extensive testing validates the efficacy and robustness of \textsc{FaST}'s core components, showcasing its potential to advance the development of cognitive visual agents in AI systems.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Multimodal Large Language Model System2 Thinking Language Agent
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/afa6dad7696adcd08f9a4feea95be266b0c566cb.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/dc8dcf4e40a4a0b9efb69511c115abe103500846.zip
Visual Agents as Fast and Slow Thinkers
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission