Geometry-aware RL for Manipulation of Varying Shapes and Deformable Objects
Robotic Manipulation;Equivariance;Graph Neural Networks;Reinforcement Learning;Deformable Objects
Yes
No ethics review needed.
1. Can HEPi generalize to more complicated geometries in the real world other than simple ropes or shapes like hearts and stars, which might bring much computational complexity to graph learning?
2. How sensitive is HEPi to the perturbation of the object coordinates from the observation?
1. The paper is well-written and presents its ideas clearly with solid formulation.
2. The proposed approach of modeling robotic manipulation problems as heterogeneous graph learning problems is well-motivated and clever to unify the structure for both rigid and deformable object tasks using sub-graphs for actuators and objects.
3. The adaptation of *SE(3)* equivariant message passing networks is reasonable and suitable to exploit the geometric symmetry for improving the sample efficiency in the large 3D search space of configurations. As far as I know, HEPi is one of the first studies of equivariant policies on 3D space within a reinforcement learning setting for robotic manipulation.
4. The empirical experiments are comprehensive and sound to support the arguments. Most results are averaged over 10 seeds using
interquartile mean with 95% confidence intervals, which is also statistically robust.
The paper proposes a novel setting for representing robotic manipulation problems as heterogeneous graph learning problems. The authors introduce a graph-based policy model, *Heterogeneous Equivariant Policy (HEPi)*, featuring multiple *SE(3)* equivariant message-passing networks (EMPNs) to model smaller sub-graphs like actuators and objects. HEPi explicitly models heterogeneity by assigning distinct network parameters for each interaction type to reduce message mixing and improve expressiveness. HEPi is claimed to be the first study of equivariant policies on 3D space within a reinforcement learning setting for robotic manipulation.

The authors theoretically prove that, for HEPi any two actuator and object nodes can exchange information while the graph network with locally connected actuators and object nodes can not. This justifies their design of actuator nodes as global virtual nodes to connect all object nodes

The authors test the proposed heterogeneous graph representation with reinforcement learning for 6 rigid and deformable object tasks, including Rigid-Sliding, Rigid-Insertion, Rigid-Insertion-Two-Agents, Rope-Closing, Rope-Shaping, Cloth-Hanging. Empirical results show that the proposed approach is more sample efficient and less likely to converge on sub-optimal solutions compared with the state-of-the-art Transformer and non-heterogeneous EMPN methods.
1. The experiments are limited to simple geometric shapes like ropes, triangles, and stars, which can be easily modeled using a few nodes (< 100 in all the experiments). However, the target objects in most 3D manipulation tasks are more complex and sophisticated, e.g., cabinets or dresses, which require significantly more nodes to model. I doubt whether HEPi can still generalize to all those complicated geometries, which brings much computational complexity to graph learning.
2. The current framework assumes that the object coordinates are readily available in the observation, which is basically a simulation setting. In real applications without coordinates, such coordinates can only be extracted from other CV models in the wild. As a graph policy model, HEPi is likely to suffer from the cumulated error in the observation and fail the tasks compared to non graph learning methods.
3. Some typos:
- L17: objects -> object.
- L182: “lifts”, the left quotation mark.
- L344: There should be a blank space before HEPi.

Nevertheless, the overall idea is novel, and empirical experiments are solid enough to support the proposed setting. I recommend that the paper should be accepted.
Yes
No ethics review needed
No ethics review needed.
Thank you for the detailed appendix, which has addressed my questions about the specifics. If the authors can supplement the experiments as suggested, I would be willing to increase my score.
+ Clever Design: This paper includes several ingenious designs. 
   + The use of EMPNs The use of SE(3)-equivariant networks ensures that the model naturally possesses SE(3) generalization, thereby reducing the search space and complexity. 
   + The use of TRPL the introduction of TRPL in place of traditional PPO makes hyperparameter tuning easier, addressing a significant challenge in RL training.

+ Detailed Appendix Experimental Description: One of the major issues in RL is poor reproducibility. However, in this paper, the authors provide a detailed appendix that lists experimental information, including the reward function and hyperparameters. This makes the paper highly reproducible.

+ Benchmark Design: This paper provides a detailed demonstration of the proposed benchmark tasks in the video and clearly defines these tasks in the appendix. As a result, these tasks and environments can be easily adopted by the research community.

I am inclined to accept this paper.
The authors propose a novel SE(3) equivariant RL method called HEPi, and introduce a new benchmark for future geometry-aware RL evaluations.  The paper use EMPNs as algorithm backbone to allow model generalize between poses and its heterogeneity. The paper use graph as representation and do thorough mathmetical analyze.
+ Real-World Application: As the authors note, this paper uses ground-truth model inputs and does not incorporate a physical robot, which means it cannot be directly applied to real-world scenarios. However, given the significant contributions of this paper in terms of benchmarking and methodology, I do not consider this a critical issue. Nonetheless, I still recommend that the authors include some specific experiments to evaluate this aspect. I suggest author can simply add visual capture pipeline. The details are as follows:
    + For task like rigid insertion and rigid sliding, you can simply add camera to take picture then use a common pose estimation module to estimate the position. The potential errors in pose estimation can be used to test the robustness of the proposed method to input errors.
    + For tasks involving fabrics or ropes, you can directly use a camera to capture point clouds and then construct a graph using the point cloud data.
+ Geometry aware: This paper does not seem to have any special designs focused on geometry, although graphs do play a role. However, the primary focus of the paper appears to be on SE(3)-equivariant designs.
Yes
No ethics review needed.
Could the authors expand more on line 51-52 about attention making the optimization landscape more difficult to traverse?

Would the policy be able to generalize to objects with different physical properties?
The proposed heterogeneous graph representation is a general representation for both rigid and deformable object manipulation tasks, and captures the geometric structure of the object. HEPi is a novel formulation of graph-based equivariant policy build on top of the heterogeneous graph representation that enables rl for manipulation of diverse shapes and deformable objects.

The paper has demonstrated thorough evaluations and ablations in simulation, with convincing results proving that the proposed method has better performance, sample-efficiency, and generalizability. Detailed discussions are provided for the results, providing interesting insights.
This paper addresses the task of manipulating diverse rigid and deformable objects with geometry-aware rl policy.

A heterogeneous graph is proposed to represent rigid and deformable object manipulation tasks. To leverage geometric symmetry for better task performance and sample efficiency, a heterogeneous equivariant policy that utilizes SE(3) equivariant message passing networks is proposed. 

Evaluation is carried out on a self-curated rl benchmark, including rigid insertion of diverse objects, as well as rope and cloth manipulation with multiple end-effectors. The proposed method outperforms baseline methods in terms of average returns, sample efficiency, and generalizability.
The object model that contains vertices are required for the object node representation, which is the main observation for the policy. This might not be easy to get in the real world due to a lot of self-occlusions during deformable object manipulation.

The paper demonstrates thorough evaluations across various simulation benchmarks, but lacks real-world evaluations to make the method fully convincing in terms of it's usefulness to real-world robot applications.

typo at line 74: equivariacne->equivariance.
Yes
No ethics review needed.
Please see the weaknesses above.
In general, I find the story in this paper intuitive and convincing. While the two core building blocks (heterogeneous graphs and equivariant policies) have been explored in prior arts, this paper presents a proper combination of both in the setting of rigid/deformable object manipulation. I feel the proposed method has the potential to be extended to manipulate fluid and rigid/soft/fluid-coupled systems, and it would be a pleasant surprise if the paper could demonstrate a fluid scene similar to the “Pour Water” task in Lin et al 2020. Of course, I understand this is outside the scope of the current submission.
The paper trains policies for one or two end effectors to manipulate rigid and deformable objects (ropes and cloths). It models the actuator and object of interest with a heterogeneous graph and advocates for a heterogeneous equivariant policy (HEPi). The paper benchmarks its method on tasks including manipulating rigid and deformable objects in IsaacLab and reports its superior performance over typical baselines.
I generally agree with the limitations listed in the paper. I think the paper can be improved in the following ways:

1. While the technical method and the story look promising, the benchmark scenes are still relatively simple because the setup only models a point-like end effector, ignoring kinematic and dynamic constraints in practical robotic hand/arm structures. Therefore, transferring the result to a real-world robot is not straightforward and probably requires more algorithmic development.

2. From a dynamic perspective, the 2D rigid-sliding task is too trivial to be a valuable benchmark for evaluating this paper and its baselines. This is already reflected in the result: “HEPi and Transformer policies perform comparably, suggesting that the limited task complexity does not fully leverage the benefits of equivariant constraints. ” I suggest this experiment remove the suction gripper and try pushing the rigid object to its target position and orientation with one/two end effectors and their contact with the object, similar to the setup in http://sain.csail.mit.edu/.

3. Deformable objects can be classified by their dimensions: codimension-2 (e.g., ropes), codimension-1 (e.g., cloths), and codimension-0 (e.g., rubber balls). The paper seems to consider the first two categories only. Handling full-dimension soft objects in the proposed method may be non-trivial because it requires observations of object coordinates and cannot be easily obtained by “integrating state-of-the-art computer vision techniques to extract key points from cameras.” I suggest the paper clarify this point by being more precise with the concept of deformable objects it deals with.
Geometry-aware RL with heterogeneous SE(3) equivariant back-bone policy for robotic manipulation
@inproceedings{
anonymous2024geometryaware,
title={Geometry-aware {RL} for Manipulation of Varying Shapes and Deformable Objects},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=7BLXhmWvwF},
note={under review}
}
Manipulating objects with varying geometries and deformable objects is a major challenge in robotics. Tasks such as insertion with different objects or cloth hanging require precise control and effective modelling of complex dynamics. In this work, we frame this problem through the lens of a heterogeneous graph that comprises smaller sub-graphs, such as actuators and objects, accompanied by different edge types describing their interactions. This graph representation serves as a unified structure for both rigid and deformable objects tasks, and can be extended further to tasks comprising multiple actuators. To evaluate this setup, we present a novel and challenging reinforcement learning benchmark, including rigid insertion of diverse objects, as well as rope and cloth manipulation with multiple end-effectors. These tasks present a large search space, as both the initial and target configurations are uniformly sampled in 3D space. To address this issue, we propose a novel graph-based policy model, dubbed Heterogeneous Equivariant Policy (HEPi), utilizing $SE(3)$ equivariant message passing networks as the main backbone to exploit the geometric symmetry. In addition, by modeling explicit heterogeneity, HEPi can outperform Transformer-based and non-heterogeneous equivariant policies in terms of average returns, sample efficiency, and generalization to unseen objects.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Robotic Manipulation Equivariance Graph Neural Networks Reinforcement Learning Deformable Objects
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/69f62640ba9d75c84e1f503d6c98e7cd2f82426e.pdf
reinforcement learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/6128c930c23fa1ef72051024c1db5dd8341cd266.zip
Geometry-aware RL for Manipulation of Varying Shapes and Deformable Objects
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission