EMOS: Embodiment-aware Heterogeneous Multi-robot Operating System with LLM Agents
Embodied Artificial Intelligence;LLM Multi-agent System;Multi-robot System;Task Planning
Yes
No ethics review needed.
I don't have any questions.
While the paper is well-written and explores an interesting application, it feels more like a project report than a research contribution, as it lacks any new machine learning methods, theoretical models, or mathematical innovations.  In essence, it describes a system with multiple robots, each equipped with an LLM and distinct capabilities. The robots communicate to determine task distribution, using robot resumes to identify which robot is best suited for each task. Through hierarchical and decentralized planning, they collaborate to complete complex tasks in a shared environment, leveraging spatial reasoning and embodiment awareness to enhance coordination.
This paper presents the Embodiment-Aware Heterogeneous Multi-Robot Operating System (EMOS), an LLM-driven, multi-agent system designed to manage diverse robots in complex household tasks.
Based on the contributions mentioned in the paper
1) The main contribution appears to center around how questions are passed to the pre-trained LLM for task allocation in multi-robot system. While this is an interesting application of LLMs, it falls short of introducing fundamentally new methods or algorithms. Rather than solely focusing on prompt engineering or question-passing techniques, the authors could explore adapting the LLM architecture specifically for the multi-robot system's needs. For example, a custom fine-tuning approach that allows the LLM to handle complex, multi-agent task decomposition and task dependencies uniquely suited to heterogeneous robots would provide more significant technical innovation.
2) Habitat-MAS simulation: It is a modification of the Habitat framework; the contribution is more of a system engineering effort than a machine learning breakthrough. 
3) Presenting results is not a contribution. Instead, it serves as a validation that the proposed methodology works as intended and is better than baselines. 
Based on the first two contributions, I would recommend the authors to submit it for a Robotics conference. Really sorry, I have to give a reject.
Yes
No ethics review needed.
1）In the domain of heterogeneous multi-agents, how does the proposed framework's use of robot resumes improve performance compared to traditional role-playing methods? Moreover, does the framework consistently outperform others across various scenarios and different robot configurations? 
2）All robot are completely interconnected, and they are collaborative under complete information. Can the proposed framework be extended to scenarios where there is a game relationship between heterogeneous individuals.
3)  What is the scalability of the framework? How does the increase in the scale of collaborative robots affect the success rate of tasks?
4） The format of resumes is artificially defined. Will the format of resumes affect the efficiency of robot assistance? How to generate a better formatted resume?
Robots’ resumes are generated by the LLM based on urdf files, which enhance the communication among robots. This paper more effectively leverages the intelligence of LLMs in addressing the challenges of heterogeneous multi-robot collaboration. LLM can not only play a planner role but also promote the communication effectiveness among robots.
This paper introduces a novel framework for controlling heterogeneous multi-agent collaboration, named EMOS. This framework integrates recent advancements in large language models (LLMs) and multimodal models, enabling intelligent cooperation among multiple robots. Unlike previous works, which primarily relied on role-playing approaches, this multi-agent framework innovatively proposes a robot resume based on the robot's URDF files. This enhancement allows for a more precise description of the robots' capabilities, thereby facilitating more efficient communication and collaboration among them. Additionally, the authors present a new benchmark named Habitat-MAS. This benchmark validates the framework's effectiveness. The paper also provides several specific experimental results, which have been published on a dedicated website.
The experimental section of the paper requires additional enhancements. While the authors conducted ablation studies on their framework and provided detailed results, there is a notable absence of comparisons with similar frameworks.
Yes
No ethics review needed.
-- See above
-- See above.
The paper presents an interesting approach to LLM-based multi-robot systems where robot roles are not predefined but are instead determined dynamically based on the robots' physical capabilities. The authors introduce the concept of a "robot resume," generated from robot URDF files and kinematic data, to guide task planning and integrate role flexibility within the model.

This concept is very compelling; however, my main concern is the additional degree(s) of freedom introduced into the decision-making and planning. In systems with predefined/assumed heterogeneous roles, the answer to “who does what” is partially predetermined, simplifying the planning process. Here, this isn’t the case, and the current centralized hierarchical LLM-based planning and task assignment system becomes significantly constrained. With the current design, the centralized planner and task assignment modules will face considerable burden in optimizing plans and assignments, particularly as domain complexity increases or when dealing with similar or more universally capable robots. I suggest comparing your approach against systems with predefined roles to illustrate any potential advantages or limitations. Additionally, you can provide a quantitative analysis of how your system's performance scales with increasing numbers of robots, robot classes, or task complexity (see [1] for example).

[1] Seraj et al. "Learning efficient diverse communication for cooperative heterogeneous teaming", AAMAS 2022

One potential solution to mitigate this issue could involve incorporating prior knowledge to streamline planning and decision-making. For example, a standard quadcopter lacks object manipulation capabilities, so it can be automatically excluded from relevant tasks, reducing the computational load. Such prior knowledge could be represented as a context/class vector for a contextual classification. Additionally, other factors such as robot availability (i.e., whether a robot is still executing a prior task) could contribute to a truly asynchronous multi-robot system.

I also have a few questions:
- Can you discuss the modularity of this approach? For instance, can you describe the process for adding a new capability like battery life to the robot resume and how it would be integrated into the task planning and allocation process. This would give a clearer picture of the system's modularity and extensibility.
- Do LLM-based planners consider assignment risks, and if so, how can these risks be leveraged in the decision-making process?
- It remains unclear how the central planner optimizes "which robot does what." For example, how does the system handle scenarios with (1) multiple equally capable robots, (2) differently capable robots that can all execute a task, or (3) universally capable robots? Does the LLM-based “discussion” incorporate strategic negotiation or cognitive hierarchy mechanisms (e.g., k-level thinking) to optimize plans and assignments? Can you provide a specific example scenario for each of these cases and explain how the system would handle the allocation in each case?
- Given the dynamic nature of the environment, assuming full observability is unrealistic. However, it is also unclear how partial observability is handled in this system. Can you discuss how your system might be adapted to handle partial observability, and what challenges you anticipate in doing so?

Additionally, I suggest including a section on Heterogeneous Multi-Robot Systems literature rather than the current Multi-Robot Systems section. Given the context of this work, it would be more impactful to emphasize on the heterogeneity in collaborative multi-robot teams, categorizing and introducing previous approaches such as MARL (heterogeneous multi-robot communication and coordination), Multi-Agent Apprenticeship Learning (learning heterogeneous multi-robot policies from human demonstrations), trait-based heterogenous multi-robot planning, and other non-LLM-based strategies. Below are a list of papers to begin with:

[1] "Learning efficient diverse communication for cooperative heterogeneous teaming", AAMAS 2022

[2] "Heterogeneous Multi-Robot Reinforcement Learning", AAMAS 2022

[3] "STRATA: A Unified Framework for Task Assignments in Large Teams of Heterogeneous Agents", JAAMAS 2020

[4] "Mixed-initiative multiagent apprenticeship learning for human training of robot teams", NeurIPS 2023

[5] "Heterogeneous policy networks for composite robot team communication and coordination", T-RO 2024

Lastly, a minor but important point: the paper contains numerous grammatical and verbal errors. A thorough review is recommended. 

Overall, I'm happy with the contributions made in this paper and vote for a weak accept. I'd be happy to raise my score given mine and my fellow reviewers' comments are adequately addressed. Thanks!
-- See above
Yes
No ethics review needed.
I suggest the authors reorganize the paper, abstract the model and ideas, and then discuss it from theoretical and practical perspectives, 
such as how to formularize the capabilities of heterogeneous multi-robot systems to satisfy various tasks' requirements, how to define the specific roles and relationships in the group, how to organize their behaviors or strategies to optimize system performances, etc.  

I suggest the authors check and refer the paper as below:

1) Yang, Q., & Parasuraman, R. Hierarchical needs based self-adaptive framework for cooperative multi-robot system. In 2020 ieee international conference on systems, man, and cybernetics (smc) (pp. 2991–2998). IEEE.

2) H. Hamann and H. Wo ̈rn, “A framework of space–time continuous models for algorithm design in swarm robotics,” Swarm Intelligence, vol. 2, no. 2-4, pp. 209–239, 2008.
This paper introduced a heterogeneous multi-robot system framework to enable effective collaboration among heterogeneous robots with varying embodiments and capabilities. It implements the proposed LLM-based multi-agent framework in indoor household environments with four tasks. The paper presents the LLM-based MAS framework conducting embodiment-aware reasoning in simulation.
This paper proposed a heterogeneous multi-robot system framework to enable effective collaboration among heterogeneous robots with varying embodiments and capabilities. However, from my perspective, the topic of this paper is too big, making it more like a technical instruction or report rather than a research paper. Moreover,  many details are not clearly explained, such as low-level planning and control (collision avoiding), mid-level decision-making, and high-level learning. Moreover, the communication graph is an important part of the MAS design, which is also not mentioned in the framework. I suggest the authors reorganize the paper, abstract the model, and then discuss it from theoretical and practical perspectives.
From my perspective, the topic of this paper is too big, making it more like a technical instruction or report rather than a research paper. Many details are not clearly explained, such as low-level planning and control (collision avoiding), mid-level decision-making, and high-level learning. Moreover, the communication graph is an important part of the MAS design, which is also not mentioned in the framework. Furthermore, several robots' plannings have apparent errors in the simulation video. For example, at the 16 seconds of the video, the drone does not follow the routine but goes through the wall directly, etc. I hope the authors can improve it and consider more details about different modules in the framework design.
A novel multi-agent framework is introduced for effective collaboration among heterogeneous multi-robot systems with embodiment-aware reasoning capabilities by understanding robot URDF files, plus a novel benchmark for this task.
@inproceedings{
anonymous2024emos,
title={{EMOS}: Embodiment-aware Heterogeneous Multi-robot Operating System with {LLM} Agents},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=Ey8KcabBpB},
note={under review}
}
Heterogeneous multi-robot systems (HMRS) have emerged as a powerful ap-
proach for tackling complex tasks that single robots cannot manage alone. Current
large-language-model-based multi-agent systems (LLM-based MAS) have shown
success in areas like software development and operating systems, but applying
these systems to robot control presents unique challenges. In particular, the ca-
pabilities of each agent in a multi-robot system are inherently tied to the physical
composition of the robots, rather than predefined roles. To address this issue,
we introduce a novel multi-agent framework designed to enable effective collab-
oration among heterogeneous robots with varying embodiments and capabilities,
along with a new benchmark named Habitat-MAS. One of our key designs is
Robot Resume: Instead of adopting human-designed role play, we propose a self-
prompted approach, where agents comprehend robot URDF files and call robot
kinematics tools to generate descriptions of their physics capabilities to guide
their behavior in task planning and action execution. The Habitat-MAS bench-
mark is designed to assess how a multi-agent framework handles tasks that require
embodiment-aware reasoning, which includes 1) manipulation, 2) perception, 3)
navigation, and 4) comprehensive multi-floor object rearrangement. The experi-
mental results indicate that the robot’s resume and the hierarchical design of our
multi-agent system are essential for the effective operation of the heterogeneous
multi-robot system within this intricate problem context.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Embodied Artificial Intelligence LLM Multi-agent System Multi-robot System Task Planning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/c09fc9fe2782771942bd7317769d91fa8f67e39c.pdf
applications to robotics, autonomy, planning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/a007787fc2f9875626c802c09c084d9e43cdf36c.zip
EMOS: Embodiment-aware Heterogeneous Multi-robot Operating System with LLM Agents
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission