GEVRM: Goal-Expressive Video Generation Model For Robust Visual Manipulation
Robot Manipulation; Vision Language Action Model
Yes
No ethics review needed.
- Baseline (video generation): is AVDC/GR-1/GEVRM trained with data augmentation?
- Baseline (CALVIN rollout): how does GEVRM compare to HiP/UniPi/GR-1/behavioral cloning with data augmentation?
- Ablation: how does setting the state alignment loss $\lambda = 0$ affect task success rate? Quantitative results in addition to the t-SNE plot would be convincing.
- Ablation/Clarification: video generation for control and goal-conditioned behavioral cloning are relatively well-studied problems. Could you clarify the key contribution of GEVRM? (How is GEVRM different from prior work in its architecture, and what design choices contributes to the improved performance?) A more comprehensive ablation studying the importance of each component in GEVRM would be convincing to see.
- Video generation for goal-conditioned behavioral cloning is well-motivated.
- GEVRM outperforms previous methods on Bridge and CALVIN for video generation.
- GEVRM outperforms HiP, UniPi, and GR-1 on generalization in CALVIN, and outperforms SuSIE on generalization with visual perturbations.
This paper introduces GEVRM, a method for robust visual manipulation through video generation. It trains a video generator for goal generation, and a goal-conditioned behavioral cloning policy. The video generator takes a sequence of observations and a language instruction, and predicts the subsequent image observations as the goal. The policy head is a goal-conditioned diffusion policy. GEVRM uses an auxiliary state alignment loss to improve learned representations. Results show that GEVRM outperforms previous methods on Bridge and CALVIN data, and is more robust to perturbations and generalizable than baselines in the simulated environment CALVIN.
- Limited/inconsistent evaluation. GEVRM is compared with AVDC on Bridge and with GR-1 on CALVIN for video generation, with HiP/UniPi/GR-1 on unperturbed generalization, and with SuSIE on perturbed generalization. It would be good to see GEVRM compared with a consistent set of baselines for video generation and policy rollouts in a few more environments.
- If the core problem setting is robust manipulation, it would be convincing to see GEVRM compared with HiP, UniPi, GR-1, and pure behavioral cloning, each with data augmentation.
- It is unclear what the key contributions of GEVRM are and why it outperforms prior work. A more comprehensive ablation with downstream task success rate could justify each component in the design of GEVRM and clarify its improvement over prior state of the art.
Yes
No ethics review needed.
1. The integration of a video diffusion model might introduce considerable computational overhead. However, the paper lacks a comprehensive analysis of the computational requirements. How about the inference efficiency in Calvin simulation environment?
2. On the goal-conditioned policy, the authors propose a diffusion policy conditioned on the generated goals. In such goal-conditioned policy framework, does the diffusion policy exhibit significant superiority compared to a simple MLP?
3. In Table 3, the generalization experiments are conducted in perturbed environments. How do the other baseline methods, apart from Susie, perform in this setting?
1. The method appears to be reasonably designed. It utilizes a video diffusion model as a visual planner to generate future predictions that guide the policy in action output. In terms of detailed design, the proposed method enhances the generation consistency through efficient video spatiotemporal compression and random mask strategies. Furthermore, the authors utilize prototypical contrastive learning to align the goal states with its current state, thereby enabling the model to implicitly infer and distinguish perturbations from the external environment.
2. The authors propose generalization experiments on perturbed environments in CALVIN (train A, B, C → perturbed test D), which can assist the research community in examining the the robustness of policies against external environmental disturbances.
This paper presents a goal-guided policy conditioned on future predictions generated by a video diffusion model. The authors try to integrate the classic principle of internal model control (IMC) into visuomotor control, thereby enhancing the robot's capability to resist environmental perturbations.
1. This paper did not undertake real-world robotic experiments. Instead, it focused exclusively on analyzing video prediction quality using the Bridge dataset and conducting action execution tests within the simulated environment of the Calvin benchmark. However, the effectiveness of GEVRM on real-world robots was not validated.
2. In Table 2, the authors compare their proposed GEVRM method with other baseline approaches within the CALVIN ABC-D environment, utilizing the settings of RGB input only. However, they did not involve powerful baseline methods for comparison. While GR-1 is a comparatively strong baseline, its reproduced performance is notably low. Could the authors provide further details regarding the reproduction of GR-1? Furthermore, would it be possible for the authors to incorporate additional powerful methods for comparison by adjusting their input modalities to RGB only? Additionally, I would like to mention that Susie's results on Calvin ABC-D should be included in Table 2.
3. The authors should give more credits to the paper "Closed-Loop Visuomotor Control with Generative Expectation for Robotic Manipulation" (https://arxiv.org/abs/2409.09016). The design philosophy behind GEVRM and the presentation of the article, such as the layout in Figure 1, seem to be inspired by this paper; however, it is not mentioned in the introduction or related works sections.
Yes
No ethics review needed.
Please see my questions above.
The paper considers an important topic in VLM for robotics. The paper is well-written, and the logic is clear. The proposed approach outperforms the baseline in terms of goal generation when the input image is perturbated and in terms of closed-loop eval in simulations.
This paper considers the problem of vision-language-action model for robotics. The proposed approach leverages a text-video generation model to generate the goal images conditioned the current observation and task description. Then the action policy leverages the generated goal and the current state to predict the robot action. To aggress the problem of disturbance encountered during deployment, the paper leverages the idea of IMC from classic control by using internal embeddings (optimized through prototype contrastive learning) as the policy model input.
The connection to IMC. The IMC in control uses “diff” between the predicted output from the process model and the actual output from the actual robot, induced by the same action, to generate the feedback signal to compensate the controller. But the proposed approach seems to use the current robot state and the desired output to generate the control signal. I think the proposed structure is just a goal conditioned policy, and I don’t see the strong connection to the IMC.


Image encoder. The image encoder is trained through the discriminating if a goal image and the current state image are from the same trajectory. The experiment is missing ablation studies on the encoder. Would be interested in how an off-the shelf encoder like DinoV2 performs.
Yes
No ethics review needed.
1. When GEVRM uses prototypical contrastive learning for state alignment, the positive pairs become the current and goal states from the same trajectory, instead of different augmented views from the same image. I wonder if this contrastive learning process will learn to focus on only trivial consistent features in the background, which are likely to be similar across all timesteps, and ignore some differences (e.g. the position change of objects or the robot arm) between the current and goal states that are essential to help the model make correct decisions.
2. Would the input states be perturbed by any transformations while training the state alignment encoders? If not, are these encoders directly used to process the perturbed images during inference?
3. During the testing time in CALVIN evaluation, apart from the visual discrepancy of the environments, are the instructions for each (sub)task seen during training, or are novel instructions tested to demonstrate task generalization capabilities?
1. This paper is well-motivated, clearly written, and easy to follow.
2. The paper properly discussed the related work that leverages text-conditioned video models to generate visual plans for decision-making.
3. GEVRM innovatively uses several techniques to improve the quality of generated visual goals and aligns state representations to enhance the robustness under perturbations
4. The authors provide extensive evaluation results on both behavior planning quality and task completion in perturbed environments, showing the effectiveness of the proposed method.
This paper introduces GEVRM, a novel close-loop VLA model that adopts the Internal Model Control principle to a learning-based framework and improves robustness to perturbations from external environments. This method leverages text-conditioned video generation to infer image goal states, aligns the current and goal states through prototypical contrast learning, and performs goal-guided action prediction using a diffusion policy. Evaluation results demonstrate the strong generalization capability of image goal generation across simulated and real-world datasets, as well as the robustness through high task success rates under external perturbations in the visual manipulation task.
1. The underlying framework – composing a text-conditioned video model and an action predictor has been widely explored by previous work, in which most works adopt a relatively lightweight method to extract inverse dynamics. This work employs a diffusion policy for action prediction and interacts with the environment in a close-loop manner, which may help learn multi-modal action distribution but can be time-consuming and often not desirable in real-time deployment.
2. The thoroughness of behavior planner evaluation can be further improved, for example, the effectiveness of techniques like random masking compared to simply masking out the last few frames for decision-making scenarios, should be supported by ablations.
3. It would be great if the authors could thoroughly discuss the limitations of the proposed method.
We propose a novel closed-loop VLA method GEVRM that integrates the internal model control principle to enhance the robustness of robot visual manipulation.
@inproceedings{
anonymous2024gevrm,
title={{GEVRM}: Goal-Expressive Video Generation Model For Robust Visual Manipulation},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=hPWWXpCaJ7},
note={under review}
}
With the rapid development of embodied artificial intelligence, significant progress has been made in vision-language-action (VLA) models for general robot decision-making. However, the majority of existing VLAs fail to account for the inevitable external perturbations encountered during deployment. These perturbations introduce unforeseen state information to the VLA, resulting in inaccurate actions and consequently, a significant decline in generalization performance. The classic internal model control (IMC) principle demonstrates that a closed-loop system with an internal model that includes external input signals can accurately track the reference input and effectively offset the disturbance. We propose a novel closed-loop VLA method GEVRM that integrates the IMC principle to enhance the robustness of robot visual manipulation. The text-guided video generation model in GEVRM can generate highly expressive future visual planning goals. Simultaneously, we evaluate perturbations by simulating responses, which are called internal embeddings and optimized through prototype contrastive learning. This allows the model to implicitly infer and distinguish perturbations from the external environment. The proposed GEVRM achieves state-of-the-art performance on both standard and perturbed CALVIN benchmarks and shows significant improvements in realistic robot tasks.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Robot Manipulation; Vision Language Action Model
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/353eb96a20b364fcb31aefae6e5a2e2d22728363.pdf
applications to robotics, autonomy, planning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
GEVRM: Goal-Expressive Video Generation Model For Robust Visual Manipulation
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission