Embodied Scene Cloning: Solving Generalization in Embodied AI via Visual-Prompt Image Editing
Embodied Intelligence;Data Augmentation
Yes
No ethics review needed.
Some introductions on the baseline GAug are appreciated.
1. The paper is well written and easy to read.
2. The visual prompt based augmentation is interesting and novel.
3. The experiments demonstrate superior results over the baseline on CALVIN.
This paper targets solving the visual gap in generalizing robot tasks. The main motivation is to augment an image in embodied scenes which leads to the technique called Embodied Scene Cloning. A visual prompt is used to augment the object or the background environment in data. Extensive experiments demonstrate the proposed augmentation has achieved performance improvements in visuomotor baselines on CALVIN. A few real-world experiments are given.
1. The task setting sounds somehow tricky. An instance image is required for unseen task deployment. How to get the specific visual prompt without background？
2. The proposed technique is limited to tasks with similar trajectory patterns, only differing in visual appearance. I am not sure how to work in a truly unseen environment. Can the system still work when other physical attributes like geometry or weight change?
3. Following the above, the author only conducts experiments on CALVIN of patterned objects and tasks. No evidence on boarder robotic tasks is given.
4. The real-world experiment is weak. Only two sequences of real-world trajectories are given.
Yes
No ethics review needed.
1. I have some questions about the authors' real-world experiments. The success rates in the table are 0%, 20%, and 60%. Could the authors clarify how many trials were conducted to determine these success rates? If the number of trials was relatively low, could the authors increase the number of experiments to obtain more accurate success rates and better validate the effectiveness of the method?
2. Could the authors increase the variety of tasks in the real-world experiments? Currently,the real-world experiments only involve the robotic arm clamping cubes from a table and placing them inside a basket. Could the authors expand the objects being grasped (e.g. type, color...) and add different task types to better validate the effectiveness of Embodied Scene Cloning in real-world scenarios?
3. The formula in "Progressive Masked Fusion" seems to have some issues. First, for the formula $\tilde{z_t} = M_t \cdot z_t + (1 - M_t) \cdot \tilde{z_t}$​, according to Fig. 2, it seems that $\tilde{z_{t-1}}$should be computed from $z_t$​ and $\tilde{z_t}$​. Maybe the formula should be changed to ​$\tilde{z_{t-1}} = M_t \cdot z_t + (1 - M_t) \cdot \tilde{z_t}$​? Second, how is $M_0$​, the initial mask, obtained?
4. I have some doubts about the experiment of Embodied Scene Cloning dataset sizes (Table 3). For some tasks, fewer amount of cloning data may have higher success rate, why would this happen?
5. The authors only conducted experiments on the CALVIN dataset. Could the authors test the effectiveness of Embodied Scene Cloning on other datasets and tasks, such as RT-1, RT-2 and BEHAVIOR-1K?
1. The experimental results on CALVIN Benchmark are promising. The results of Embodied Scene Cloning are significantly higher than the baseline, demonstrating that this method is effective and can effectively improve the model’s generalization ability to unseen foreground and background.
2. The goal of this paper is to address the generalization problem on test tasks, which is crucial for embodied AI. The authors provide some insights on this issue.
This paper aims to enhance the generalization of robotic policies to new environments by visually "cloning" source demonstrations using scene-specific visual prompts. 
To tackle the generalization challenge, it introduces a new framework, Embodied Scene Cloning that utilizes image-editing techniques to generate environment-aligned trajectories. 
By maintaining coherence between edited and non-edited regions, the model achieves smooth, consistent visual elements that better represent target deployment scenarios, both forground and background. 
Empirical results show significant performance improvements over existing methods, such as GreenAug, in simulated environments, indicating the approach's effectiveness in augmenting training data for policy generalization. 
The authors also conduct real-world experiments to showcase that the framework has the potential for real-world application.
1. Technique Contribution is limited. This work is a combination and application of Diffusion Model, DDIM, SAM2, and Depth-Anything V2, without a novel techniqle contribution.
2. The real-world experiments are insufficient. The authors conducted extensive experiments on the CALVIN benchmark, validating the method’s effectiveness in this setting. Although they included real-world experiments, the results are rather limited. The experiments focus solely on the foreground scenario with a single setup, addressing only the object-grasping task. The authors should consider expanding the real-world experiments to further verify the generalization capability of the proposed framework.
3. The practicality of the Embodied Scene Cloning method is not very high. In real applications, manual observation and comparison of the differences between the training and test environments are required,(e.g. identify the blue box to yellow box transition) followed by the manual design of visual and text prompts based on these differences, which introduces additional complexity. If there are differences in both the background and multiple objects between the test and training environments, this method will be difficult to apply, limiting the scope of the pipeline’s usability.
4. Minor concerns:
There are some typos and mistakes in the paper that need to be addressed:

     (i) In section 4.2, Line 332-334 Among the five research questions, the third one, “As the amount of cloned data... ”  is a statement, not a question. The authors should change that into a proper question.

     (ii) In section 4.2, Line 428-430 check this sentence, it’s been repeated.

     (iii) In section 4.2, Line 421, 427  The authors use three patterns of reference to the table, namely “Table 3”, “Table III”, “Table.4”. The authors are encouraged to change these into unified reference.

     (iiii) In section 4.2, Table 3. For lift_table task, there are two bold value.

     The authors should check grammar, typos, and phrasing to improve the quality of the paper.
Yes
No ethics review needed.
See details in the weaknesses.
1 The proposed approach provides a potential solution to seamlessly integrate of new scene elements while maintaining the consistency and fidelity of the background areas. This will inspire the community to design visually-controlled data augmentation strategies in the future.

2 The effectiveness of the proposed approach is demonstrated in both simulation and real-world environments.
This paper proposes a novel data augmentation method called Embodied Scene Cloning for addressing the generalization challenge in robotic manipulation. The proposed method incorporates Conditional Feature Extraction, Progressive Masked Fusion, and Visual-Prompt Guided Image Editing to fulfill visual-instructed source demonstration cloning. As a result, the proposed method allows precise visual control over both the generated objects and the background environments compared to text-prompt-based data augmentation methods. Experimental results in both simulation and real-world environments demonstrate the superiority of the proposed approach over the state-of-the-art generative data augmentation methods, especially GreenAug.
1 My main concern is the actual generalization ability of the proposed method in real-world applications and the fairness of comparing the proposed method with existing methods. Generally, it is defaulted that the agent in the trained environment cannot access any information in unseen environments. And augmentation for a specific new embodied scene may not be a scalable way of generalization since the unseen scenarios are infinite. Moreover, the proposed policy seems to be relatively effective for texture or color generalization for objects, but its generalization to more common situations such as objects of different types, stacks, and occlusions is not verified. 

2 Additionally, the proposed method mainly focuses on the robotic manipulation task while the title claims a large topic by “Solving Generalization in Embodied AI”. I suggest the author give a more precise claim in both the title and the main text.

3 There lack the ablation studies to analyze the effect of specific design for different method components. Moreover, the failure cases should be given for a more complete analysis of the proposed approach.

4 Can the proposed method support sim2real augmentation? The additional experiments on this are appreciated. It will further demonstrate the generalization ability and practicality of the proposed approach. 

5 Visual prompting is a widely used strategy in both VLMs and text-to-image generation approaches. The authors should add an independent subsection to analyze existing methods and discuss the comparison in the Related Work section.
Yes
No ethics review needed.
1. The papers refers "traditional augmentation methods" as text based. Meanwhile, there are other traditional augmentation such as color jitter, random crop etc. Suggest directly refer them as text based. In fact, it would be nice see if simple color jitter improve the performance.
2. In line 323, it says including baselines of RoboFlamingGo(Li et al., 2023) and 3D Diffusion Actor. Meanwhile, they are policy learning method not augmentation method. Therefore, it's confusing to call them baselines (which should be GreenAug for example).
3. More papers in image translation for robotics should be discussed in related works. For example: 
Li, Yiwei et al. “ALDM-Grasping: Diffusion-aided Zero-Shot Sim-to-Real Transfer for Robot Grasping.” ArXiv abs/2403.11459 (2024): n. pag.
Originality: applying diffusion based image editing to change foreground background in robotics
Quality: good ablation study on elements of environment
Clarity: explaining context methods
Significance: improve policy domain transfer
The paper introduces a visual-prompt-based framework that enhances policy generalization in robotics by prompting with visual cues from deployment environments. It clones scenes with visual alignment, improving performance in novel environments.
The main weakness is the tested environment is relatively simple. Only changing the color/texture of one object in foreground / background. To demonstrate real potential, needs more extensive experiments.
@inproceedings{
anonymous2024embodied,
title={Embodied Scene Cloning: Solving Generalization in Embodied {AI} via Visual-Prompt Image Editing},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=dZbCoATni7},
note={under review}
}
Recent advancements in robotic learning have enabled robots to perform a wide range of tasks. However, generalizing policies from training environments to deployment environments remains a major challenge, and improving these policies by collecting and annotating demonstrations in target environments is both costly and time-consuming. To address this issue, we propose Embodied Scene Cloning, a novel visual-prompt-based framework that generates visual-aligned trajectories from existing data by leveraging visual cues from the specific deployment environment. This approach minimizes the impact of environmental discrepancies on policy performance. Unlike traditional embodied augmentation methods that rely on text prompts, we propose to "clone" source demonstrations into the target environment and edit it with visual prompt to effectively improve the generalization ability on specific embodied scene. Experimental results demonstrate that samples generated by Embodied Scene Cloning significantly enhance the generalization ability of policies in the target deployment environments, representing a meaningful advancement in embodied data augmentation.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Embodied Intelligence Data Augmentation
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/597b578d4a2c710400916b52a1c926a38cc3a3bd.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Embodied Scene Cloning: Solving Generalization in Embodied AI via Visual-Prompt Image Editing
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission