Do Think Tags Really Help LLMs Plan? A Critical Evaluation of ReAct-Style Prompting
Large Language Models;ReAct;Sequential Decision-Making
Yes
No ethics review needed.
N/A
It conducts a very detailed sensitivity study into the effectiveness of ReAct prompting.
This paper studies into what makes ReAct-type of prompting works. It studies into 3 factors: (1) where the guidance is provided, (2) the different types and structure of this guidance (3) on varying the resemblance of example prompt to the queried problem.
Studies on prompt engineering and what parts of prompt work for final result has been studied for a long time. Various research on why and what types of intermediate thinking chains/in-context learning can work (such as "Rethinking the Role of Demonstrations") and on what models they work have been studied by many papers. However, no such papers are cited or discussed.

Many of the observational conclusions such as similarity to in-context examples and uselessness of thinking trace are also discussed by various papers, and thus renders this paper's conclusion not so exciting.
Yes
No ethics review needed.
1. In Table 1, the explanation for GPT-3.5 outperforming GPT-4o is “highlighting the brittleness of claims of ReAct.” I don’t quite understand this rationale, as it seems to imply that weaker prompting leads to more significant performance drops in stronger models. Shouldn’t stronger LLMs be more robust to suboptimal prompting? Could you provide some examples to clarify this counterintuitive result?

2. The Webshop dataset shows a low accuracy. I believe analyzing such low accuracy is not that meaningful. Other papers report much higher figures. Can you explain the reason?
1. This paper provides a valuable exploration of how ReAct functions in decision-making tasks.

2. I highly appreciate the motivation for existing work, which holds great importance for the research community.
This paper explores the impact of different components of ReAct-style prompting on the final outcomes, showing that both interleaved reasoning traces and generated reasoning traces have minimal influence on results. In contrast, the similarity between examples has a much more significant effect.
1. The writing in the paper is difficult to follow. Prompts take up too much space in each paragraph, and some words are struck through without clear explanations. The font size in the figures is too small. I am confused about the names "GPT-3.5-instruct" and "GPT-3.5-Turbo" as the authors refer to "gpt-3.5-turbo-0125" and "gpt-3.5-turbo-instruct". I also struggled to understand what "Act" is in the tables 1 as it's hard to find the introduction of baselines. Overall, the paper is not ready for publication.

2. The examples provided, such as finding an item, do not seem to have sufficient changes in the environment. The authors suggest that giving LLMs a complete action plan at the outset (e.g., if A happens, do X; if B happens, do Y) is feasible, as shown in Figure 2. I disagree as many situations could change the environment, making it impossible to provide all potential scenarios upfront.

3. Regarding RQ1 and its two variants, I didn't find any intrinsic difference between these variants and React. I feel that the variants are just human-rewritten versions of React.  A more reasonable comparison would be to directly present the user’s query to the model and let it generate a similar plan prompt. The current setup, where the authors design prompts based on potential behaviors, seems unfair.

4. The conclusion that the "similarity between the example and the query" is important feels rather obvious. Since LLMs rarely encounter agentic structure data in their training, the significance of in-context learning is naturally high. This conclusion lacks sufficient insight for me.
Yes
N/A
No ethics review needed.
Regarding scalability: Could you clarify how your findings apply to more complex environments or tasks beyond AlfWorld and WebShop? For instance, would the same dependency on example-query similarity hold in domains with less structured actions or greater variability?

Regarding justification of approaches: In your experiments, how much do you attribute the performance outcomes to the prompting method versus the underlying capabilities of the LLMs themselves?

Regarding workflow completeness: Did you explore how the size of the context window or the amount of training data provided to the LLMs impacts performance in your scenarios? While the analysis of ReAct’s limitations is thorough, do you have suggestions or possible remedies for reducing the reliance on example-query similarity?
Originality: The paper offers a new look at ReAct-style prompting by questioning if interleaving reasoning with actions truly improves LLM performance. It highlights example-query similarity as the main factor driving success, offering a different perspective.

Quality: The paper is well-structured, with clear experiments tested across different LLM models and domains. The use of varied tests and detailed analysis strengthens the reliability of the findings. The overall experiment setups are convincing and comprehensive.

Clarity: The paper is easy to understand and well-organized. Proposed ideas are reasonable, and tables and figures effectively explain the results, making the findings accessible to a wide audience.

Significance: The paper raises important questions about ReAct-style prompting, showing that example-query similarity may play a bigger role than reasoning in LLM performance. This insight could help guide future research and improve LLM prompting methods.
This paper examines how effective ReAct-style prompting is for improving Large Language Models (LLMs) in decision-making tasks. While ReAct is known for boosting LLM reasoning by combining reasoning and actions, this study questions where these improvements really come from. The authors conduct experiments in planning tasks in AlfWorld and WebShop to analyze the effects of reasoning traces, plan guidance, and how similar examples are to the queries.

Contributions: The paper tests ReAct's claims and finds that LLM performance doesn't significantly improve by manipulating reasoning with action execution. The results show that the main reason for LLM performance is the high similarity between example tasks and queries, rather than reasoning abilities from reasoning traces. A detailed sensitivity analysis shows that LLM performance drops when the example-query similarity is reduced, highlighting ReAct's fragility. The study challenges the belief that LLMs develop reasoning abilities through ReAct-style prompting, showing that success is mostly due to pattern matching or retrieving similar examples.
While the paper offers important insights into the limitations of ReAct-style prompting, it doesn’t fully address whether these findings apply across different scenarios and domains. The study focuses on specific tasks in AlfWorld and WebShop, but it’s unclear how generalizable the results are to other environments or more complex tasks. For example, would the same reliance on example-query similarity hold in tasks with more diverse or less structured action spaces? The lack of broader applicability raises concerns about the scalability of the conclusions, making it hard to know if the findings can be generalized to all situations where ReAct prompting is used.

The study uses black-box LLMs without fully clarifying what matters more for performance—the ReAct-style prompting itself or the inherent capabilities of the LLMs. Since LLMs differ in architecture and training, it's difficult to separate the effects of the prompt from the underlying model’s abilities. This makes it unclear whether the performance issues are caused by the prompting technique or limitations within the models themselves. Ideally, more transparency in how the LLMs are interacting with the prompts and what exactly drives the observed behavior would help strengthen the conclusions.

The paper also lacks a strong technical contribution beyond critiquing ReAct. There is no obvious novel method or solution offered to address the identified weaknesses in ReAct-style prompting. While finding flaws in prompting techniques is important, the absence of a proposed solution limits the paper’s impact. Readers are left with an understanding of what is wrong but without a clear path to improve or resolve these issues. A more balanced approach would include recommendations or a framework for improving prompting techniques, which would provide more concrete value to readers.
Yes
No ethics review needed.
See Weaknesses.
This work studies a wide range of settings to understand how different components of the ReAct prompts affect the model's performance. The results are insightful for researchers and practitioners to better interpret what ReAct prompts are doing and the current LLMs' reasoning/planning performance.
This paper examines the impact of different components of ReAct-style prompting for planning problems. The authors propose several kinds of alterations to the ReAct prompts, and find that 1) the LLM's performance is minimally influenced by the interleaved reasoning trace or the content of the reasoning trace in the prompt; 2) the performance heavily depends on a high degree of similarity between the demonstrations and queried problems. These findings indicate that LLM's reasoning/planning abilities are more fragile than normally perceived.
- **Incomplete analysis for supporting the claims**. The paper does not include enough details/analyses about what the LLMs' actual generations are like and their comparison with the (altered) demonstrations. Are LLMs following the changed demonstrations when addressing new problems, or are they still doing things in the original ReAct style? For example, is it possible that the LLMs are still doing interleaved reasoning and acting, even though the demonstrations are changed to have all thought steps together in the beginning? For cases where the performance drops a lot (e.g., the Domain, Instance variations), are these new errors caused by the model's decreased reasoning abilities, or simple mistakes around the surface-form symbols? Relatedly, the authors often make claims that are somewhat ambiguous on the target. For example, in lines 416-418: "...the performance of LLMs either improved or remained consistent when provided with weaker or irrelevant guidance information. This refutes ReAct’s claim that task-specific reasoning trace is the source of LLM agent performance". Is this "task-specific reasoning trace" the ones in demonstrations or those generated by the model? The results only show that LLMs don't need such reasoning traces in the demonstrations, but the LLMs could still generate good traces during inference.

- **Results are overall not surprising**. It is well known that LLMs are usually not "learning" how to perform the task from the demonstration examples, rather, the prompt mostly provides the overall format and some key anchors such as label/action space related to the test problems to shape the model generation [1, 2]. There are prior works showing that one doesn't even need to provide these explicit demonstrations for the model to work, e.g., just saying "think step by step" could elicit CoT behaviors of LLMs. It is also well-known that providing examples that are more similar to the queried problems brings better performance, and many prior efforts on demonstration selection are exactly about closing the gap between the demonstrations and queries, e.g., [3, 4]. LLMs, or ML models more broadly, generally suffer from distributional shifts which is one of the open research problems. Reporting this in some specific tasks/settings is not very significant in my view.

--

Citations
- [1] Min et al. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? EMNLP-22.
- [2] Wang et al. Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters. ACL-23.
- [3] Liu et al. What Makes Good In-Context Examples for GPT-3? DeeLIO-22.
- [4] Rubin et al. Learning To Retrieve Prompts for In-Context Learning. NAACL-22.
We study ReAct for sequential decision-making problems, and contrary to the original claims, show that the reason for improvement stems from high exemplar similarity with the query problem and not due to the generated & interleaved reasoning traces.
@inproceedings{
anonymous2024do,
title={Do Think Tags Really Help {LLM}s Plan? A Critical Evaluation of ReAct-Style Prompting},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=85Ik12q2hP},
note={under review}
}
The reasoning abilities of Large Language Models (LLMs) remain a topic of debate, which are critically tested in sequential decision-making problems. ReAct, a recently popular method has gained popularity for claiming to enhance LLM reasoning abilities while directly prompting them by $``\textit{interleaving reasoning trace with action execution}"$ in text-based planning domains such as AlfWorld and WebShop. However, given the different components of ReAct-style prompting, it remains unclear what the source of improvement in LLM performance is. In this paper, we critically examine the claims of ReAct-style prompting for sequential decision-making problems. By introducing systematic variations to the input prompt, we perform a sensitivity analysis along the original claims of ReAct. Contrary to these claims and common use-cases that utilize ReAct-style prompting, we find that the performance is minimally influenced by the interleaved reasoning trace or by the content of these generated reasoning traces. Instead, the performance of LLMs is primarily driven by the unreasonably high degree of similarity between input example tasks and queries, implicitly forcing the prompt designer to provide instance-specific examples which significantly increases the cognitive burden on the human. Our empirical results, on the same suite of domains as ReAct, show that the perceived reasoning abilities of LLMs stem from the exemplar-query similarity and approximate retrieval rather than any inherent reasoning abilities.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Large Language Models ReAct Sequential Decision-Making
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/6001cad8f32517bd1044c1b2acceffffa42897ed.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/005ca68c35e657b4076bcaf8db2e5d37eed8a0e3.zip
Do Think Tags Really Help LLMs Plan? A Critical Evaluation of ReAct-Style Prompting
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission