Closed Loop Interactive Embodied Reasoning for Robot Manipulation
Michal Nazarczuk Jan Kristof Behrens Karla Stepanova Matej Hoffmann Krystian Mikolajczyk


IEEEexample:BSTcontrol

[
    John B. Little
    April 29, 2024
==================

empty
empty


Embodied reasoning systems integrate robotic hardware and cognitive processes to perform complex tasks typically in response to a natural language query about a specific physical environment. This usually involves changing the belief about the scene or physically interacting and changing the scene (Sort the objects from lightest to heaviest). 
In order to facilitate the development of such systems we introduce a new simulating environment that makes use of MuJoCo physics engine and high-quality renderer Blender to provide realistic visual observations that are also accurate to the physical state of the scene. 
Together with the simulator we propose a new benchmark composed of 10 classes of multi-step reasoning scenarios that require simultaneous visual and physical measurements. 
Finally, we develop a new modular  Closed Loop Interactive Reasoning (CLIER) approach that takes into account the measurements of non-visual object properties, changes in the scene caused by external disturbances as well as uncertain outcomes of robotic actions.
We extensively evaluate our reasoning approach in simulation and in the real world manipulation tasks with a success rate above 76% and 64%, respectively. 





§ INTRODUCTION



The research efforts in developing systems capable of high-level perception and reasoning <cit.> have been increasing in recent years.
Many robotics systems include an agent that is required to perform a task with a specified goal given visual observations or instructions in natural language  <cit.>. Such tasks requires a simulation environment for generating data, or on-line agent training, together with an evaluation process. Existing environments <cit.> differ in terms of physics simulation engines and the quality of visual observations.
High computational load of realistic rendering forces a compromise between real-time physics calculation and visual quality of the simulation. 
Realistic synthetic visual data are crucial for development of robotics systems but the collection of such data is often restricted by a cumbersome setup process and real time robot operations <cit.>.
More powerful simulators, datasets and benchmarks are needed for a closed-loop setup where changes to the world or physical measurements (, weight, elasticity, occlusions, etc.) should be accounted for as directly affecting the next step in action planning.



In this paper, we emphasise the following contributions[The environment, the benchmark and the reasoning approach are released on the project webpage https://michaal94.github.io/CLIERhttps://michaal94.github.io/CLIER]:

∘ We introduce , a novel modular environment for simulating and executing interactive manipulation tasks characterised by high-quality visual rendering and accurate physics calculation. It employs MuJoCo for physics and Blender for rendering, which we refer to with  (MuJoCo+Blender Environment). It can generate multimodal data based on scene specification and instruction templates. An example scene and task that can be executed in this environment is shown in Fig. <ref>. We believe to be the first work focusing on task planning in robotics manipulation while preserving accurate physics modelling between the bodies, in contrast to discrete action spaces such as in ManipulaThor <cit.> and ALFRED <cit.>.  

∘ We propose , a new benchmark  for evaluating embodied closed loop reasoning systems for robotic manipulation.  It includes 10 types of benchmarking tasks for single- and multi-step tabletop manipulations that require combined reasoning between visual observations (recognising attributes of objects and their relations) and physical measurements (acquiring properties obtainable through interaction only, such as weight or stiffness). extends <cit.> with a challenging combination of various modalities (language, vision, and manipulation) for benchmarking Visual Reasoning, VQA (<cit.>) (language + vision), and Embodied question answering (EQA) (<cit.>) (language + vision + navigation) approaches.

∘ We propose a closed-loop neuro-symbolic interactive reasoning approach (), that can plan,  evaluate and  execute a given task (, Place the heaviest mug on the box). The highlight of our approach is that it can plan for perception of invisible object properties (, weight or stiffness) and account for new measurements to adjust the plan. Our reasoning agent is able to recover from a wide range of manipulation failures caused by external disturbances and uncertain outcomes of manipulation actions. 

∘ We evaluate our proposed Closed Loop Interactive Embodied Reasoning () within using the proposed synthetic benchmark and validate it via comparative experiments between simulated and real-world   YCB <cit.> objects. 

In the remainder of this paper, we first review the existing simulators and justify the need for a new environment. We then present the proposed and in Sec. <ref>. Sec. <ref> introduces our benchmark. Results and discussion are presented in Sec. <ref>.



§ RELATED WORK






Simulation environments for manipulation focused on combined visual and language robotic tasks. State-of-the-art simulators ManipulaTHOR <cit.> (extension of AI2-THOR <cit.>), CoppeliaSim <cit.>, or iGibson <cit.>  provide  non-photorealistic rendering, lacking accurate shades and reflections. This is in contrast to Blender that can generate photorealistic scenes with procedural object models. ISAACSim <cit.> is a resource-intensive, closed-source simulator with limited customisation. ThreeDWorld <cit.> also makes use of a commercial renderer and does not provide the infrastructure for robotics manipulation tasks, including motion planners. Unlike prior works, we pay particular attention to conditional action planning and execution in robotics manipulation while preserving accurate physics modelling between the bodies, and offer both continuous and discrete action spaces, in contrast to discrete action spaces such as in ThreeDWorld <cit.> and ManipulaThor <cit.>, or pure continuous control of the robot as seen in Robosuite <cit.>. For example, ManipulaThor <cit.> considers grasping as a discrete action, while we enable to execute sequences of various custom primitive actions (e.g., approach, gripper closing, lifting, etc.) or continuous control of the robot. From these simulators, only iGibson offers non-kinematic continuous and discrete object states temperature or burned. Our MuBlE environment differs from the aforementioned manipulator simulators by a unique and open-source combination of 1) high-quality visual data and realistic physics modelling, which is a key advantage when attempting sim2real transfers (as shown in our experiments), 2) ability to select between operating on continuous action space or usage of primitive actions, and 3) considering both visible and non-visible continuous object states (e.g., weight, or stiffness). We present a comparison between different  simulators in Table <ref>. 

Benchmarks for language-conditioned manipulation. The advantages of MuBlE environment enabled us to prepare a unique benchmark for closed-loop embodied reasoning (SHOP-VRB2) that extends SHOP-VRB <cit.>.
SHOP-VRB is a dataset closely related to the robotic manipulation scenarios considered in this work. It features various objects suitable for robotic manipulation that can be easily obtained. There are many high quality 3D models of various instances of these objects which make them suitable for generating synthetic scenes via Blender rendering that includes procedurally generated materials rather than simply texturing the mesh. We therefore extend it to a dataset and a benchmark SHOP-VRB2 that enables development and testing of systems capable of reasoning on non visual attributes and performing interactive tasks.Compared to other benchmarks, such as ALFRED <cit.> (based on RoboTHOR), a benchmark for understanding natural language instructions in robotics, VLMbench <cit.>, based on RLbench <cit.> and CoppeliaSim <cit.>, that adds linguistic commands and realises automatic  complex tasks builders, IKEA Furniture Assembly <cit.>, that provides a benchmark for various assembly tasks and CALVIN benchmark <cit.> offering natural language instructions for long-horizon manipulation tasks, our SHOP-VRB2 benchmark requires long-horizon closed-loop embodied reasoning, that involves manipulation of the world to acquire knowledge about non-visual object properties in order to complete the task or answer a query.

Symbolic approaches gained interest in visual reasoning tasks and several deep learning systems incorporate neuro-symbolic programs. CLEVR-IEP <cit.> suggested predicting symbolic programs to be executed on disentangled scene representation, and was  followed by NS-VQA <cit.>, and NS-CL <cit.>. V2A <cit.> introduced a symbolic approach for robotic tasks using only the initial state of the scene, and was evaluated in a real robotic environment <cit.>, however, only open-loop single-step tasks were demonstrated. Symbolic approaches were also considered in Task and Motion Planning systems (TAMP) that were often  based on the formal language PDDL <cit.>. Classic TAMP approaches rely on predefined rules and known dynamic models <cit.> but learning based methods are increasingly replacing handcrafted heuristics in TAMP methods <cit.>. PDDLStream <cit.> extends PDDL to add any sub-symbolic model in a black-box way, which also opens it to the use of learning-based methods. However, it works only on a symbolic level and does not close the loop with the real world. Deep Visual Reasoning <cit.>  directly predicts task plans by considering initial observation only. Regression Planning Networks (RPN) <cit.> propose a task planning model which performs regression in symbolic space using neural networks. RPN are expanded in <cit.>   by utilising regression planning on scene graph representations to estimate the next subgoal based on the current scene and a given scene graph, solving only subpart of the overall task that our method  deals with. The interaction with the real world that follows a linguistic query (measure the weight of the mug, find the sofa) is explored in embodied question answering approaches <cit.>. In <cit.>, language-conditioned visual reasoning is combined with manipulation. Navigation of agents based on vision and language was investigated in VLN <cit.>.  However, both approaches considered tasks that do not react to incoming measurements, reasoning is not conditioned on new observations.

Recently, several works utilised large-langauge models (LLMs) to generate action plans for robot manipulation tasks (eg.,  <cit.>, <cit.>). While they show impressive versatility and adaptability to novel environments, they suffer from lack of explainability, verifiability, and repeatability. LLMs require careful prompt engineering and filtering of outputs before the robot can execute them. Furthermore, such approaches do not capture the task and trajectory details and do not enable fast-loop reactiveness. These problems have to be resolved to broadly deploy LLMs for embodied reasoning. Using data created with environment, we train and evaluate a novel reasoning agent that can act on a natural language query in a real scene. Moreover, we present closed-loop reasoning experiments in simulation and real tabletop scenes. A highlight of our approach is that it plans for perception of invisible object properties, when stacking objects by weight. Moreover, unlike classical methods, our reasoning agent is executed on every key frame, which allows it to recover from a wide range of manipulation failures (e.g., unsuccessful grasping, or falling object during manipulation).



§ MUBLE ENVIRONMENT AND CLIER REASONING



In this section, we present our   environment for simulating manipulation tasks and reasoning approach that makes use of this environment. The main modules and data flow are presented in Fig. <ref>.

 is built on 𝚛𝚘𝚋𝚘𝚜𝚞𝚒𝚝𝚎 <cit.> which is a simulation framework suitable for creating robotic environments inside the physics engine MuJoCo. We equip our environment with high quality rendering powered by Blender. It is designed for a generic tabletop scenario with a single robotic manipulator and a gripper. We use the same set of robots and grippers as 𝚛𝚘𝚋𝚘𝚜𝚞𝚒𝚝𝚎. We provide a set of object models in MuJoCo and their counterparts in Blender along with templates to create new items.

We first introduce the scene graph in Sec. <ref> and action planning in Sec. <ref> that is a part of . We then present two main components of the environment and reasoning approach: action (render) loop in Sec. <ref>, and physics loop in Sec. <ref>. Sec. <ref> summarises our reasoning pipeline.





 §.§ Scene Graph


A real scene can be captured by a camera or a synthetic one can be rendered by Blender based on a detailed scene description. Scene graph 𝙶 is generated for each key frame 𝙸 starting from the first image of the scene.   A keyframe is rendered after a motion corresponding to the primitive action is finished e.g. moving from object 1 to object 2, approaching grasping pose, closing the gripper, etc. We have chosen the keyframe based approach as it allows us to use Blender for rendering frames, which have sufficient quality for  sim2real transfer, and sufficient speed to allow for recovering from failures with the closed loop approach. 
 
To parse the scene into a set of objects with attributes 𝙶̃ we apply an instance segmentation with Mask R-CNN <cit.>, followed by classification of object attributes with ResNet <cit.> and pose regression with  CosyPose <cit.>. Scene graph is a sequence of feature vectors corresponding to each object in the scene including whether the object is in the gripper or whether the gripper is closed. We use simple geometrical heuristics to create or update the scene graph 𝙶 for keyframe 𝙸, similarly to <cit.>. 

High-quality rendering of every frame can be used for training but it is too slow for real-time experiments. However, many tasks are achievable with low rate visual reasoning and fast feedback loops from other sources of information (force control). Example key frames rendered during execution of a task are presented in Fig. <ref> (Left).



 §.§ Symbolic program generator and action planner



As the reasoning method we propose a multi-staged approach inspired by Neural Symbolic VQA approaches <cit.>.  Symbolic program generator is implemented as a Seq2Seq network <cit.> that given translates the natural language query 𝚃 into a sequence of symbolic programs that are evaluated on the scene graph 𝙶 for keyframe 𝙸, producing a subgoal 𝚂 for the task measuring weight. We employ a transformer as the action planner to predict the next primitive action 𝙰 and its target, given the current scene graph 𝙶 and subgoal 𝚂. 

To enable planning of various manipulation tasks we design a set of primitive actions 𝙰 that can be easily extended in :

    
  * 𝚖𝚘𝚟𝚎 – moves the end effector towards a given target (another object or part of the table),
    
  * 𝚊𝚙𝚙𝚛𝚘𝚊𝚌𝚑 – positions the end effector in a grasping position with respect to the target object
    
  * 𝚌𝚕𝚘𝚜𝚎_ 𝚐𝚛𝚒𝚙𝚙𝚎𝚛, 𝚘𝚙𝚎𝚗_ 𝚐𝚛𝚒𝚙𝚙𝚎𝚛 – to grasp or release,
    
  * 𝚕𝚒𝚏𝚝, 𝚕𝚘𝚠𝚎𝚛 – lifts or lowers the end effector,
    
  * 𝚠𝚎𝚒𝚐𝚑 – weighs the object in the gripper,
    
  * 𝚜𝚚𝚞𝚎𝚎𝚣𝚎 – squeezes the object in the gripper to measure its stiffness.

Non visual states, such as weight or stiffness are encoded in the scene graph after measurements, however the transformer is provided only a binary flag indicating whether  the property was measured or not and the sorting reasoning is performed once all the measurements are completed.



 §.§ Action (render) loop



Scene graph and action planner make part of the action (renderer) loop in Fig. <ref>.  Primitive action controller implements a control that can execute action 𝙰 on target object in MuJoCo physics engine or real robot. Physics loop, discussed in more details in  Sec. <ref>), calculates physics and collects observations 𝙿 (e.g. pose of end effector, physical measurement, force in the gripper, etc.). A control signal is generated in every step of the physics loop, until the path is completed. After the execution, a new key frame is captured in real-world setup or rendered in simulation.

Primitive action execution with obstacle avoidance require the positions and orientations 𝙿 of the objects in the scene. Every primitive action, including gripper closure, generates a desired trajectory (position and orientation) 𝙿 for the end effector. The action of approaching to the grasp position entails a set of pre-coded grasp sequences. We implement grasping of cylindrical objects (depending on their position and size), grasping by the edge, grasping by the handle etc. We use a simplified model as simulated grasping is often not reliable and inconsistent with  real grasping. Whenever grasping pads on all gripper fingers are in contact with the same object, it is considered as grasped. Thereafter, the object's pose is fixed with respect to the end effector. Further, whenever the gripper starts to open, the object is detached from the gripper body with an initial velocity of 0. More details can be found on GitHub page: <https://michaal94.github.io/CLIER>



 §.§ Physics loop



The physics are calculated with the use of MuJoCo engine which makes part of and is used by as shown in Fig. <ref>. With a user defined timestep, the control signal 𝙲 is applied to the manipulator, the corresponding forces are calculated and applied to all objects in the scene. Similarly to 𝚛𝚘𝚋𝚘𝚜𝚞𝚒𝚝𝚎, the control signal takes the form of the displacement of the end effector, position and orientation change, along with binary signal for opening and closing of the gripper. By default, the control signal is translated to desired joint forces by the Operational Space Controller <cit.>, which can be customised. A control signal to the environment can be provided by the user directly or can be obtained from the primitive action controller, which calculates the error of current end effector pose and the planned trajectory.

Every step of applying the control 𝙲 affects the physics of the scene and generates a set of observations 𝙿, which include: position and orientation of the end effector, joint positions and velocities, status of the gripper closure. Observations of measured non-visual properties, such as stiffness, weight, or elasticity, are also included.  Additionally, measurements related to all objects in the scene may be collected, such as positions, orientations, bounding boxes, indication whether the object is currently in the gripper. Any custom sensor can be added to as in 𝚛𝚘𝚋𝚘𝚜𝚞𝚒𝚝𝚎.



 §.§ Reasoning pipeline



 starts by parsing the visual scene to create or update its semantics and geometry scene graph. Simultaneously, a symbolic program in the CLEVR-IEP <cit.> format is extracted from a natural language instruction. The program is then executed on the scene graph to select the subgoals that have to be achieved on the scene, measuring the weight of the target when the graph node of the target's weight is empty. Further, action planning regresses the task to a sequence of primitive actions for the current state of the scene. It reevaluates the next step at every pass of the action (render) loop. The reasoning is completed when either the answer is obtained from the program, or the state of the scene matches the target in task completion modules. The components of the pipeline  are presented in Fig. <ref> and an example task in Fig. <ref>.


 





§ DATASET
 

We introduce dataset created in for training and benchmarking. The dataset includes a set of scenes with instructions to perform various tasks  (e.g. Stack metal objects from heaviest to lightest). Tasks are designed to enforce reasoning simultaneously on the visual observations (recognising attributes of objects and their relations) and continuous physical measurements, taking the feedback loop into account. Every example is accompanied with a ground truth sequence of actions for successful execution, along with visual observations and detailed scene graphs. Some examples can be found in Fig. <ref> and <ref>. We include weight measurement as a representative example of estimating non visual object properties through manipulation due to the ease of repeatability for other researchers but we also demonstrate stiffness measurements. Other properties such as roughness are also possible but require another template for training and an implementation of the control.  
We first discuss the tools we provide in that can be used to generate a scene and natural language instructions Sec. <ref>. We then introduce our benchmark in Sec. <ref>.



 §.§ Data generation tools



Scene generator
is a tool in that can procedurally generate data for training models. We provide a template scene that is compatible with the renderer within the framework. The procedural algorithm takes a set of object models, which can be customised, and randomly places them on the tabletop. Desired properties of the objects such as colour, material, and size can also be randomised.  We check for collisions between meshes, unlike previous approaches that check only bounding boxes. Afterwards, a maximal desired level of occlusion is ensured.  Finally, we render an image of the scene, generating full ground truth, including segmentation masks for all the objects, the robot, and the tabletop, as well as a depth map. Fig. <ref> (Right) presents some example scenes generated with . Objects in the scene are described in Sec. <ref>.

Instruction generator

is another tool in that given the scene, generates natural language instructions or tasks that require reasoning and interaction. We propose a template-based algorithm where we improve upon CLEVR and SHOP-VRB, which are time-consuming and scale poorly when more adjectives are introduced in the templates. This is due to evaluating all possible combinations of describing words, and validating the descriptions with respect to the given scene. Instead, we implement an efficient rejection mechanism.  We first generate all combinations of short descriptions for all scene objects. Short descriptions of objects are more natural than in <cit.>. Next, we evaluate the descriptions using scene constraints the target of the instruction has to satisfy the constraint of being pickupable otherwise the description is rejected. The instruction targets are randomly selected from the validated descriptors and used to generate ground truth symbolic program. We automatically produce the ground truth actions by reasoning backwards with predefined pairwise relations lifting an object requires grasping it,  grasping requires approaching to the grasping position, etc.). Fig. <ref> presents example instructions generated for the given scenes using templates that are described in Sec. <ref>. 



 §.§ Benchmark data



scenes include 12000 realistically rendered scenes 
split to train, validation, and test sets: 10000:1000:1000. The scenes contain typical household objects which are easily accessible. Following <cit.>, various instances of objects are included: baking trays, bowls, chopping boards, food containers, glasses, mugs, plates, soda cans, thermoses, and wine glasses. The scenes are generated with 4 to 5 objects per scene (their respective distribution is 47.3% and 52.7%). We make sure the randomization of the materials (plastic, metal, glass, rubber, wood) is realistic. Out of 54317 placed objects, a subset was selected randomly, with 1330 instances of the least common (big chopping board model due to many possible collisions), and 3486 instances of the most common model (bowl – less prone to collisions).

YCB scenes include 30 simulated benchmarking scenes with  9 YCB-Video <cit.> objects and 3 randomly generated scenes for each of the benchmarking tasks described below. 9 YCB objects were selected so that the objects share various types of visual/physical attributes (cleanser, mustard, mug, bowl, tomato can, Cheez-it, sugar box, meat can, foam brick).



Benchmarking tasks  We assign one instruction to each scene to introduce more diversity in visual observations. We designed 10 classes of benchmarking tasks revolving around moving and stacking objects based on their visual (colour, material, shape), and physical properties (weight). Example templates are presented in Tab. <ref>. The task types are closely related to the instruction templates in Tab. <ref> and involve 1) measuring weight of a single object, 2) measuring weight of multiple objects, 3) picking up based on weight, 4) moving single object 5) moving multiple objects 6) moving based on weight 7) stacking objects 8) stacking objects according to weight 9) stacking 3 objects 10) ordering objects according to their weight. 𝙾𝙱𝙹𝚡 refers to a description of an object consisting of a set of visual properties (chosen randomly and validated), the red object presented in Fig. <ref>. Note that tasks may refer to either one, specific unique object, or a set of objects sharing a certain property template 1 and 2 in Tab. <ref>. Further, 𝚃𝙿𝚡 specifies a part of the table the left and the right part. Finally, 𝚆𝚂𝚡 refers to weight specifier, distinction whether the lightest or the heaviest object from the set is the target of the instruction. The length of the instructions ranges between 5 and 16 words. The resulting sequences contain between 5 (measure the weight of the single object) and 46 (stacking several items according to weight) primitive actions. All instructions are accompanied with symbolic programs in CLEVR-IEP <cit.> format and task specifications with respect to the scene graph.

Ground truth for visual and physical observations are provided for every scene and its benchmarking instruction. An observation is taken after every primitive action is executed and contains: position and orientation of end effector, gripper status, weight measurement, ground truth scene graph, segmentation masks, and action with its target.

 




§ BENCHMARKING AND RESULTS



In this section we provide benchmarking results for  .  We show results for two datasets: 1) in simulation for dataset (Sec. <ref>), and 2) on a set of 30 benchmarking scenes in simulation and in real-world with YCB objects (Sec. <ref>). 



Benchmarking metrics
are the rate of successful task execution (stacking) and correct question answer (weight query). Additionally, we provide accuracy split into task types presented in Tab. <ref>. Finally, for the inference tool of we provide a classification of incorrect attempts, including errors of execution, incorrect scene recognition, loop detection, timeout, inconsistency in object tracking. 

The latency of various modules is the following. CosyPose inference is  0.8s, attributes recognition:  0.15s, transformer action planner:  0.02s (as run on 2080Ti). The prediction of the next action takes around 1s as measured on the hardware. These are reasonable delays given that these modules are deployed at keyframes.



 §.§ experiments



Results for  are presented in Table <ref>(Left) which shows success rates for tasks. We observe high success rates for the tasks that include manipulation of single objects (weighing or moving one object), and strong decrease in accuracy for multi-object manipulation (stacking more objects, preceding manipulation with weighting). Further, we identify the most common reasons for failure in Table <ref>(Right) which reports the distribution of error codes from the environment. Execution error (14.4%) accounts for failures in execution of primitive actions which may arise from inaccurate scene description (typically object pose). Scene inconsistency (12.6%) refers to mistakes in tracking objects IDs between frames. Loop detection (10.8%) arises when primitive action chains are repeated (when approaching object to grasp with misaligned position). Note that this experiment uses simple, ResNet based pose estimation in contrast to  more accurate CosyPose <cit.> that was trained for YCB objects but not for objects. Finally, we believe that overall accuracy of 43.9% on indicates that this data forms a challenging benchmark for evaluating visual and interactive reasoning.



 §.§ Real-world experiments



In this section, we report comparative results for real YCB scenes that mirror 30 YCB scenes introduced in Sec. <ref>.  

Experimental Setup includes Franka Emika Panda <cit.> arm with 7 degrees of freedom and a 2-finger parallel gripper. An (extrinsically calibrated) Intel Realsense D455 camera is set to face the robot and capture the front view of the scene (see Fig. <ref>). To allow to control the real robot, we developed a thin ZMQ <cit.> based communication layer that emulates the same interface as for the simulated robot (, motion generation using position-based servoing). Special purpose skills include measuring stiffness via squeezing the object with different forces and measuring deformation or weight via lifting and joint torque differences. Note that the weighing has a relative error of less than ± 10% in the range 0-200 g compared to the ground truth weight. The stiffness measurement has a coefficient of variation ranging from 1.6-3.4 %, which is highly repeatable. However, we were not able to obtain ground truth data for stiffness.

Sim2Real Our method uses raw RGB rendered images to train the attribute recognition module (Fig. <ref>). This is not further tuned between YCB sim and real experiments. The same reasoning model (Sec. <ref>) is evaluated in 30 synthetic and real scenes using the same  CosyPose <cit.> on RGB images. The reasoning modules operate on the scene graph thus are agnostic to visual input. 



Results We mirror experiments with the real scenes in our simulation environment.  To evaluate the contribution of different modules we perform testing along with ablations by replacing them with ground truth (GT) data 1) reasoning success with   GT pose and  GT attributes 2) pose inference with GT attributes and GT reasoning, 3) pose inference and attribute recognition with GT reasoning, 4) full inference without using GT. The results for all these combinations are in Table <ref> where the success is the average across 3 runs of each experiment. Note that Seq2seq network in symbolic program generator is not included in the table as it performed with 100% accuracy, therefor the error for case 1) comes from transformer action planner. We observe a very similar performance between simulation and real which validates the usefulness of high quality vision and physics in . The performance for individual tasks with YCB scenes in simulation and real world is reported in Tab. <ref>(Left). Sim and Real differences for YCB can be attributed mostly to CosyPose object detection and its robustness to occlusion.

Typical execution errors
Our closed loop reasoning approach is able to recover from various execution errors such as dropping or moving an object (see video). Typical errors during the real world execution are caused by uncertainty in pose estimation. This leads to various cases:  1) grasp succeeds because object's real pose is withing the margin for error. 2) a different grasp is achieved over a rim instead over diameter of an open cylindrical object. 3) the object slips out when closing the gripper. 4) the gripper collides with the object during the approach and causes a robot error (safety stop reflex). The robot can recover from all except case (4) or when the object is pushed out of the workspace in (3). A varied pose of the object in the gripper (i.e., (1) and (2)) occasionally influences subsequent tasks such as stacking or weighing. Detection of occluded objects is also prone to failures. In particular, occlusion from the gripper when grasping is an issue when updating the scene graph. We use the last estimated object pose and leverage the information of the gripper holding 'something' to overcome these pose estimation errors. 

Joint limits and singularities. Robot kinematic agnostic servoing of the end-effector can lead to degenerate robot configurations. These situations include 1) joint limits, 2) the borders of the workspace, 3) self-collisions, and 4) alignment of several joint axes. 
To avoid running into joint limits, we start each experiment in a good standard pose and the robot always rotates to a neutral down-facing end-effector orientation before moving via the smallest rotation to the goal rotation. Self collisions are avoided by the Panda controller and are counted as robot errors.  





§ DISCUSSION AND CONCLUSIONS
 

In this paper, we presented our closed-loop approach for visual and interactive reasoning for robotic manipulation tasks: closed loop interactive embodied reasoning (). This approach is using our novel  environment that incorporates MuJoCo physics simulation with high-quality renderer and enables  generation of multi-modal demonstration data for robotic manipulation tasks. This fully modular environment enables both data generation and benchmarking of simultaneous reasoning in visual and physical space.  is able to incorporate observations of both visual and physical attributes of the manipulated objects into long-term reasoning. Capturing data from visual and physical measurements in the shared scene graph enables the symbolic reasoning approach and simplifies the implementation of the closed loop approach via keyframe based reasoning. The results from simulated and real-world experiments showed its ability to successfully transfer between the simulated and real environment and to recover from various errors or changes in the scene.

We believe this work provides a new challenge and a benchmark for future systems that aim at bridging the gap between physics simulation for robotic manipulators, realistic visual simulations and execution in real environments.



IEEEtran




