InteractiveCOT: Aligning Dynamic Chain-of-Thought Planning for Embodied Decision-Making
Embodied Agent;Multi-modal Large Model;Chain-of-Thought;Planning;Reinforcement Learning;Preference Learning
Yes
No ethics review needed.
- I'm wondering why the proposed method works much better on the task Heat & Place and Examine in Light, while worse on Pick & Place. Can authors provide more insights on these?

- What will the proposed model perform without COT prompting? Since the proposed method re-plan the task at interactive steps, is COT helpful in this case? I'm interested to see the ablation study with and without COT prompting.

- Intuitively, re-planning should work better when the action sequence is long, and the closer towards the end of the action sequence, the more accurate and less uncertain the planned actions should be. Does the proposed method behave like this? 

- There are typos and grammar mistakes in the text and figure, please check the spelling.
- Replacing PPO in RL4VLM with DPO can increase average success rate.
This paper used Direct Preference Optimization (DPO) with a Vision Language model that uses Chain-of-Thought (COT) prompting for task planning of embodied agents. The proposed method optimized the COT process instead of the final action as in PPO. The experiment showed the proposed method outperformed RL4VLM in terms of the average success rate on ALFWorld dataset.
- The contribution is incremental. The main contribution of this paper is replacing PPO in RL4VLM (which also has COT) [1] by DPO [2] with necessary adaptions.

- The motivation that the authors want to use DPO is that it could be beneficial in the cases of long action sequences, partial occulsion and multi-tasking. However the results do not show any of these scenarios. It's not clear whether DPO performs better thanPPO in these scenarios.

- More analyses are needed to show the performance gain on ALFWorld, details see comments below.


[1] Zhai, Yuexiang, et al. "Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning." arXiv preprint arXiv:2405.10292 (2024).
[2] Rafailov, Rafael, et al. "Direct preference optimization: Your language model is secretly a reward model." Advances in Neural Information Processing Systems 36 (2024).
Yes
No ethics review needed.
I have listed the questions in the bullet points of the weaknesses section.
How to benefit from and optimise chain of thoughts in LLM-derived models for decision-making is an interesting question which the paper considers.
The paper proposes a preference based fine-tuning method for a VLM on sequential decision making tasks. They look at Alfworld tasks and show a 6% improvement over the baseline RL4VLM (which looks at RL based finetuning of VLMs). They add auxiliary losses (APW and APC) to encourage COT - action consistency in the output form the VLM.
The paper is very poorly written:
1) Many implementation details are left to the imagination, there isn’t a code implementation provided either. 
2) The supplementary material only contains a single page derivation and no experimental details. 
3) Lots of typos and confusing wording throughout the paper
4) Lines 214, 215, what is meant by “πref refers to the policy generated by the unrefined output of VLM? Donate → Denote 
5) πθ and πref donate the policy generated by VLM - not clear enough
6) Line186  - typos? - For once(?) planning instance, planner output the …
7) Fig 1 : dicision -> decision
8) In many places there is mention of a “Preference action” - I suppose the authors mean to say “preferred action”
9) Fig 1 is not very clear - what is the circle trying to outline in the “preference action” and “not preference action”? 
10) The subscripts in the equations are not defined anywhere
11) Line 222 typo in the section heading
12) The contributions claim interactiveCOT supports both PPO and DPO- where is this validated? Isn’t the paper only presenting DPO results?
13) Are their pairs action level or trajectory level
14) Point 2 in contributions: “tailored to the interaction characteristics of embodied agents, “ - what does this mean?

Lack of thoroughness in experiments:
1) The paper doesn’t mention what exactly is the reward function used by the PPO baseline? The paper mentions using partial task progress signal (which is much denser than a 0/1 success signal indicating whether the episode was successful) to construct preference labels - did the baseline (PPO) also benefit from this extra information? If not, why? It should be possible to convert the partial progress to rescaled rewards.
2) Fig 6a both variants seem the same in terms of the max performance resached yet the text describes one variant (w action weighting) as better than the other
3) No information about the hyperparameters is provided, no information on how many seeds were used, fig 4b does not even have error bars - was only one seed used per task??
4) How was the checkpoint used to report the final score chosen? Fig 4a has a larger gap between the two methods at 2k steps (6% as claimed by the authors), but the gap would be smaller (by around 2%) if you used the ckpt for PPO at 1k training steps - which would reduce the improvement to 4%. 
5) How exactly were the trajectories collected?
6) No other baselines are studied apart from RL4VLM:  the RL4VLM paper cited studies a CNN+RL baseline, as well as a prompting-only LLM baseline. It would also make sense to compare to the baseline where the authors train a VLM with DPO directly for action selection (as done for the VLM-RL case in [1]).
7) What is the reason/motivation for using the particular preference score definition in eqn 10? Why 50 * progress  specifically?
8) 454 line - unclear, and where is this demonstrated?
9) Line 426 - extremely vague description of scores computation (“In practice, considering the achievement of long-term goals, we calculate preference scores using a method similar to discount factor weighting in reinforcement learning returns”)

[1] Large Language Models as Generalizable Policies for Embodied Tasks
Yes
No ethics review needed.
See the weaknesses.
- Utilizing foundation models as a starting point for embodied agents is an interesting approach, as it can enhance sample efficiency and reduce redundant exploration by leveraging the prior knowledge embedded in these models.
- InteractiveCOT achieves a 6% improvement in the final success rate by focusing more on optimizing the CoT process during VLM fine-tuning.
This paper introduces InteractiveCOT, a method for fine-tuning a vision-language model (LLaVA-7B) for embodied AI tasks. The authors aim to enhance RL4VLM by optimizing the chain-of-thought (CoT) process rather than focusing solely on final action decisions. Using the Direct Preference Optimization (DPO) algorithm, they fine-tune the vision-language model with online-collected episodes. The proposed method is evaluated in the ALFWorld environment, demonstrating improvements over the PPO baseline.
- The paper's presentation is difficult to follow, with numerous typos and inconsistencies in notation. For example:
    - Line 84, "In summery" -> "In summary".
    - In Figure 1, "LLaVA as dicision model" -> "LLaVA as decision model."
    - Line 215, "donate" should be "denote"; additionally, $\pi_{ref}$ is duplicated.
    - The definitions of subscripts and superscripts for action (i.e., $a_t^1$ and $a_t^2$) in line 245 and in Equations (4), (6), (7), (8), and (9) are inconsistent.
    - Line 213 references the tuple with $o_t$, but it is unclear where $s_{1 \sim t-1}$ originates.

- The authors should include a background section to introduce the basic RL framework, including elements of the MDP, trajectories, and policy, to clarify the RL context being considered. Without this, it is difficult to follow the subsequent sections. Additionally, a brief overview of the original DPO algorithm should be provided so that modifications proposed in the methods section are clearly distinguishable. 

- In Section 3.1, the authors state that the VLM is used as a planner; however, it is unclear how the plan is generated. It appears that the VLM functions directly as a policy, outputting final actions to step into the environment, as illustrated in Figure 1. Thus, it may be misleading to frame the proposed method as a "re-planning framework" (line 197). Can author clarify this point?

- What type of action space does the paper consider? Is it continuous or discrete? If it is discrete, how is the MSE calculated in Eq. (7)?

- In line 201, what does "well-fine-tuned model" refer to? Is this the VLM fine-tuned by the proposed method?

- Throughout the paper, what does $\tau_t^{t-1}$ represent?
@inproceedings{
anonymous2024interactivecot,
title={Interactive{COT}: Aligning Dynamic Chain-of-Thought Planning for Embodied Decision-Making},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=Y4iaDU4yMi},
note={under review}
}
Vision-Language Models (VLMs) are increasingly being employed as the decision-making "brains" of embodied agents. Effectively harnessing their powerful generalization capabilities in dynamic, context-specific tasks remains a significant challenge. Chain-of-Thought (CoT) prompting is often utilized for complex task execution, but existing methods either rely on static strategies that fail to adapt to changing environments or fine-tune on offline datasets, which are insufficient for optimizing agent decision-making through interaction.
In this paper, we propose a novel approach that focuses on optimizing the CoT reasoning process rather than just the final action tokens. By aligning the CoT process through preference-based reinforcement learning, specifically Direct Preference Optimization (DPO), we enhance the agent's ability to make accurate decisions in dynamic environments while mitigating model degradation during fine-tuning. Our method models the environment as a Markov decision process, requiring the agent to reflect on the current state in real time to generate adaptive plans and actions.
By prioritizing the optimization of the CoT process over the final actions, we enhance the agent's reasoning adaptability while effectively mitigating model degradation during fine-tuning.
Experiments in the ALFWorld environment demonstrate an average success rate of \textbf 26.67%, which is a 6\% improvement over RL4VLM, and show that our method effectively mitigates model degradation post fine-tuning. These results highlight the potential of integrating preference-based reinforcement learning techniques with CoT processes to enhance the decision-making capabilities of vision-language models in embodied agents.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Embodied Agent Multi-modal Large Model Chain-of-Thought Planning Reinforcement Learning Preference Learning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/d5dd1633021d19e3f816b6eb6f279400dbcec24c.pdf
applications to robotics, autonomy, planning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
InteractiveCOT: Aligning Dynamic Chain-of-Thought Planning for Embodied Decision-Making
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission