title: Indirect Object-to-Robot Pose Estimation from an External Monocular RGB Camera
authors: Jonathan Tremblay Stephen Tyree Terry Mosier Stan Birchfield
text: 
  
  
  
  Vortex Dynamics and Phase Diagram in the Electron Doped Cuprate Superconductor Pr_0.87LaCe_0.13CuO_4
      L. Ghivelder
      December 30, 2023
  ====================================================================================================
  
  empty
  empty
  
  
  
  
  
  
  
  We present a robotic grasping system that uses a single external monocular RGB camera as input.
  The object-to-robot pose is computed indirectly by combining the output of two neural networks:  one that estimates the object-to-camera pose, and another that estimates the robot-to-camera pose.
  Both networks are trained entirely on synthetic data, relying on domain randomization to bridge the sim-to-real gap.
  Because the latter network performs online camera calibration, the camera can be moved freely during execution without affecting the quality of the grasp.
  Experimental results analyze the effect of camera placement, image resolution, and pose refinement in the context of grasping several household objects.
  We also present results on a new set of 28 textured household toy grocery objects, which have been selected to be accessible to other researchers. To aid reproducibility of the research, we offer 3D scanned textured models, along with pre-trained weights for pose estimation.[Project page  https://research.nvidia.com/publication/2020-07_Indirect-Object-Poselink.  Video: <https://youtu.be/E0J91llX-ys>]
  
  
  
  
  
  
  
  § INTRODUCTION
  
  
  Determining the pose of an object with respect to the camera has received much attention in the computer vision and robotics communities <cit.>.
  However, this is only half the battle.
  To be useful in the context of a robotic manipulation system, the object's pose must be ascertained with respect to the robot.
  
  
  The standard setup involves either 1) an in-hand camera, or 2) an external camera.
  The former approach has the advantage that, once the camera's pose has been calibrated with respect to the robot's end effector, it never needs to be recalibrated (assuming that it remains rigidly attached).  
  Moreover, a camera-in-hand is able to see the object at arbitrarily close distances and multiple viewpoints, thus providing a potentially large signal-to-noise ratio to determine the object's location before grasping.
  On the other hand, a camera-in-hand has difficulty seeing the surrounding context; if, for example, the object is disturbed by an external force that moves it out of the camera's field of view, the robot must blindly recover—a potentially dangerous maneuver when the environment is dynamic and unstructured.
  As a result, camera-in-hand systems are often augmented with an external camera for sensing the robot's workspace from a global view.
  
  This reasoning leads naturally to the latter approach of primarily using an external camera for pose estimation.
  However, external cameras typically require an off-line calibration process that is notoriously tedious, sensitive, and error-prone.
  This process computes the camera's pose with respect to the robot in one or more images using fiducial markers such as ARTags <cit.> or AprilTags <cit.>.
  Such markers must be placed carefully to avoid occlusion, and their pose with respect to the robot must be known.
  Moreover, once the camera has been calibrated in this manner, the camera's pose cannot be disturbed, or the entire delicate procedure must be repeated.
  
  One of the goals of our lab is to develop a system that can reliably and robustly perform pick-and-place of known objects in its workspace using readily-available sensing technology.
  In this paper we report on recent progress toward that goal, in which we equip a robot with a single monocular RGB camera mounted on an external tripod.
  As shown in Fig. <ref>, an object's pose with respect to the robot is computed indirectly by combining the output of two neural networks:
  one that estimates the object's pose with respect to the camera <cit.>, and another that estimates the camera's pose with respect to the robot <cit.>.
  The latter network enables online calibration, so the camera can be moved freely without affecting the quality of the results.
  These networks were both developed in our lab and were trained using only synthetic data, relying on domain randomization <cit.> to bridge the sim-to-real gap.
  
  
  
  
  
  
  
  This paper contains the following contributions:
  
      
    * We present a robotic system capable of estimating the pose of known objects in the robot's coordinate system using an external monocular RGB camera with online calibration.
      
    * The accuracy of the system is analyzed, along with the effects of image resolution, camera position, pose refinement, and other system parameters.  We show that accuracy of 2 cm is achievable under a variety of conditions.
      
    * A robotic grasping system is demonstrated, showing the ability to grasp objects even when the camera is moved.  These objects include a set of YCB objects, along with a novel set of 28 readily acquired toy grocery objects which we introduce.  To aid reproducibility, we provide 3D scanned textured models and pre-trained pose-estimation weights for these new objects.
  
   
  
  
  
  
  
  § PREVIOUS WORK
  
  
  
  
  
  
  One useful way to categorize robotic grasping systems is by the type of sensor input.
  Many modern robotic manipulation systems rely on commodity RGBD sensors (Kinect, RealSense, or PrimeSense).
  
  
  One approach is to use the point cloud to refine the pose of an object once it has been detected, as in PoseCNN <cit.>. 
  
  
  A more principled approach is to learn to jointly process depth and RGB to estimate the object's pose, as in DenseFusion <cit.>. 
  The same researchers <cit.> show that it is possible to learn to extract 3D keypoint representations from RGBD data, which can then be used for robotic manipulations, pouring.
  Stevšić  <cit.> use RGBD pose estimation for an assembly task, focusing on detecting poses of known geometries in arbitrary and complex contexts.
  The promising works of Florence  <cit.> and Manuelli  <cit.> show it is possible to learn feature representations (sparse or dense, respectively) from RGBD inputs which are consistent across instances in the same category (mugs) and useful for grasping and placing objects in desired poses.
  The former approach, which operates on keypoints, shares similarities with that of our earlier work in Tremblay  <cit.>.
  
  
  
  
  
  
  Another approach is to use the point cloud from the depth camera to predict grasping positions directly, without RGB information.
  Mahler  <cit.> learn to evaluate the quality of candidate top-down grasps sampled from point clouds.
  Mousavian  <cit.> learn to sample and refine a diverse set of 6-DoF (degrees-of-freedom) grasps extracted from segmented point clouds.
   
  
  Murali  <cit.> extended this work to cluttered scenarios by learning to discriminate between successful grasps and grasps which collide with the environment. 
  
  
  Finally, a few systems rely on a single external RGB camera. 
  Tremblay  <cit.> present a method to retrieve the 6-DoF pose of known household objects from a single RGB camera which has been previously calibrated to the robot.
  The method leverages a network to predict 2D image coordinates for corners of a bounding cuboid of known dimensions, from which the pose is computed using PnP.
  Pauwels and Kragic <cit.> present an online calibration procedure that tracks objects to compute the camera-to-robot pose from multiple images.
  Other work leverages an RGB camera to learn a policy in image space to control a robotic arm for grasping. 
  For example, Tobin  <cit.>, James  <cit.>, and Rusu  <cit.> use simulation to learn neural network policies capable of directing a robotic arm to grasp colored cubes (or other simple geometric shapes) using images as input.
  These works do not separately compute the pose of the object or robot, and have demonstrated only limited tolerance to variation in camera pose or workspace configuration (dimensions of table, height of object) relative to training.
  Similar approaches have also been explored in the context of reinforcement learning <cit.>.
  To date, no work has shown the ability to grasp objects using a single RGB camera with online extrinsic calibration.
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  § METHOD
  
  
  
  In this section we describe in detail the different components of our proposed system.
  
  
  
   §.§ Estimating object-to-camera pose
  
  
  We use a deep neural network to estimate the 6-DoF pose (3D rotation and translation) of known objects with respect to the camera.
  This network, based upon our earlier work called DOPE <cit.>, consists of a multiple stage convolutional network that transforms the input RGB image into a set of belief maps, one per keypoint.
  
  Typically, n=9 keypoints are used to represent the vertices of a bounding cuboid, along with the centroid.
  In addition to the belief maps, the network outputs n-1 affinity maps, one for each non-centroid keypoint.
  Each map is a 2D field of unit vectors pointing toward the nearest object centroid.
  The maps are used by a postprocessing step to individuate objects, allowing the system to handle multiple instances of each object type.
  Pose is determined by applying a PnP algorithm <cit.> to the keypoints detected as peaks in the belief maps.
  
  The input to the network consists of 533×400 images processed by a VGG-based <cit.> feature extractor, resulting in a 50× 50×512 block of features.
  These features are processed by a series of 6 stages—each with 7 convolutional layers—which output and refine the belief maps described above.
  
   
  
  
  
  Please refer to <cit.> for further details.
  For training, we use 120K synthetic images of each object generated by our publicly-available NDDS tool <cit.>.
  For the YCB objects, the dataset was split between domain-randomized <cit.> images and photorealistic images from FAT <cit.>, as described in Tremblay  <cit.>; for the new objects, all data was domain-randomized, without photorealistic data.
  Training consisted of 60 epochs using the Adam optimizer <cit.> and a learning rate of 10^-4. 
  
  
  
   §.§ Estimating camera-to-robot pose
  
  
  The 6-DoF pose of the robot with respect to the camera is also estimated using a deep neural network.
  This network, based upon our recent work called DREAM <cit.>, consists of an encoder-decoder that transforms the input RGB image into a set of belief maps, one per keypoint.
  Because only one robot is in the scene, affinity fields are not needed.
  In this paper we focus on the Rethink Robotics Baxter bimanual robot with which our lab is equipped.
  While this robot was mentioned in Lee  <cit.>, no quantitative experiments were performed.
  This paper extends that work by analyzing the performance quantitatively in the context of object grasping, along with various system improvements that we found necessary to achieve good results in practice.
  
  One of those improvements is to augment the keypoints located at the 14 joints of the robot with 10 additional keypoints defined manually on the robot's torso (see Fig. <ref>).
  These latter keypoints are important for achieving pose stability when the arms are mostly out of the camera's field of view, which occurs when the camera is viewing the scene from a close range (less than about 1.4 m).
  The input to the network is a 400×400 image, downsampled and center cropped from the original 640×480.
  The layers of the network are as follows: the encoder follows the same layer structure as VGG, whereas the decoder 
  is constructed from 4 upsample layers followed by 2 convolution layers resulting in the keypoint belief maps.
  The network is trained using synthetic data generated using an internal version of NDDS <cit.> that allows the robot joints to be moved at random while simultaneously randomizing the camera pose.
  As with DOPE, domain randomization <cit.> is used in training to bridge the sim-to-real gap.
  In this work we also propose a new version of the architecture that uses two stages to further refine the estimates. 
  
  
  
  
  
   §.§ Estimating object-to-robot pose
  
  
  For grasping an object, it is imperative to calculate the pose of the object with respect to the robot's coordinate frame.
  This is achieved simply as:
  
      ^R_OT = (^C_RT)^-1 ^C_OT,
  
  where ^R_OT is the pose of the object in the robot frame, ^C_RT is the pose of the robot in the camera frame (calculated by the second network), and ^C_OT is the pose of the object in the camera frame (calculated by the first network).
  
  Note that in contrast to end-to-end learning <cit.>, the modular approach presented here has the advantage that the system can be repurposed without having to retrain all the networks.  
  For example, to apply our system to a new robot, only one of the networks needs to be retrained.
  Similarly, to apply the system to new objects, networks only need to be trained for those objects, independent of the robot and of other objects.
  Moreover, this modular approach facilitates testing and refinement of the individual components to ensure accuracy and reliability.
  
  
  
  
  
  
   §.§ Pose refinement
  
  
  Estimating the 6-DoF pose of an object from a monocular RGB camera is challenging, because—being an inverse problem—small errors in the input can lead to large errors in the output.
  Any network that directly outputs keypoint locations or poses will create errors in the estimate due to limited training data and network capacity.
  One way to address this issue is to apply a pose refinement step that adjusts the pose parameters by iteratively matching the input image with a synthetic projection of the model according to the current pose.
  For this step we rely upon the matching technique of Pauwels  <cit.>, which we modified by replacing the detection step with the first network above; thus only the tracking step is used for pose refinement.
  This additional processing helps to insulate measurements from slight errors caused by the networks.
  
  
  
  
  
  
  § TOY OBJECT SET
  
  
  
  
  Reproducibility and sharing solutions to accelerate the pace of research are fundamental challenges for the robotics research community.
  Given the progress in computer vision over the past decade, we aim to replace fiducial markers with unmodified, everyday objects for robotic manipulation research, thereby allowing more realistic scenarios.
  The most widely used set of objects in manipulation research today are the YCB objects <cit.>.
  However, these objects are no longer easily acquired[Even for researchers in the USA, it is difficult to purchase matching items due to lack of continued availability or significant changes in texture; in other countries the problem is even more acute.]
  and only a subset meets the size and weight constraints for manipulation with typical robotic arms.
  
  To this end, we propose a new set of objects for robotic manipulation research.
  Our design objectives dictate that the set of objects be:  1) somewhat realistic, 2) of the proper size and shape for grasping by typical robotic end effectors, and 3) accessible to researchers throughout the world.  
  One important emerging area in robotic manipulation is that of household robots for automating daily chores in, say, a kitchen.  
  Such applications are crucial for facilitating aging-in-place, assisted living, and healthcare-related challenges.
  While we initially intended to scan real items from a local grocery store, we quickly realized that this approach comes with several fundamental limitations:  1) such objects are not widely available to researchers around the world, 2) the reflective properties of many of these objects (metallic cans or reflective labels) make for difficult scanning, and 3) the texture of real-world objects frequently change due to seasonal marketing campaigns.
  
  
  
  As a result, we settled on a set of 28 toy grocery objects that are widely available online, see Fig. <ref>.
  Because the containers are hollow, there are no issues with perishability or shipping, and therefore they can be easily purchased online by anyone in the world.
  Total cost for all objects is less than 50 USD.  
  Moreover, these objects are of the proper size and shape for typical robotic grippers.
  Using an EinScan-SE scanner, we generated 3D textured models of these objects, from which we created synthetic training images for pose estimation neural networks.
  These 3D models and the trained pose estimation weights for all 28 objects are made available, both as a baseline system for comparison as well as for practical use in robotics labs.
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  § EXPERIMENTAL RESULTS
  
  
  In this section we seek to answer the following questions: 
  What absolute accuracy can the system achieve in estimating the object-to-robot pose?
  How is accuracy affected by image resolution, robot keypoints, and pose refinement strategies?
  When the camera is moved around the scene, how accurately can the system servo above an object?
  And, finally, how does object-to-robot pose accuracy translate into grasp success?
  
  
  
  
  
  
  
  
  
  
   §.§ Experimental setup
  
  
  Our experiments use a Baxter robot which has two 7-DoF (degrees of freedom) arms; 
  we use only the right arm.
  The parallel jaw gripper's range of motion is 4 cm, so pose estimates with error greater than 2 cm generally result in a failed grasp.[
  More specifically, for an upright cuboid, this threshold is only applicable along the line joining the two fingers, as the system is more forgiving in the orthogonal direction.  
  For an upright cylinder, the threshold is applicable in both directions.
  In all cases, the system is more forgiving along the approach direction (vertical in our experiments).]
  The system uses a single Logitech C920 webcam, and all processing is done on a local PC with an NVIDIA Titan Xp GPU.
  Given the reach of the arm and field of view of the camera, the robot's workspace, the volume in which graspable objects can be placed, is about 20×20×15 cm in front of the robot. 
  The camera can be placed anywhere in front of the robot within approximately 90^∘ azimuth, distance from 1 to 2 m, and height approximately the same as the torso.
  Although full 6-DoF pose estimation facilitates more complex grasping maneuvers, for this paper we limit the system to using top-down grasps for simplicity; the extension to other grasps is straightforward.
  Even so, note that grasps can occur at arbitrary height (objects are not restricted to rest on the table or any other plane).
  
  
  The robot is controlled using Riemannian Motion Policies (RMPs) <cit.> to follow pre-specified waypoints 
  (relative to the object) to move the robot to different locations.
  Both neural networks process a single image, with no tracking over time.
  As a result, for safety and stability, both object-to-camera and camera-to-robot poses are processed with an exponentially weighted filter to dampen oscillations that otherwise would occur from errors in the network outputs.
  This introduces a slight lag (on the order of 1–2 seconds) in system response.
  
  
  
   §.§ Positioning accuracy
  
  
  We assessed the accuracy of the pose estimation of the object with respect to the robot for a variety of objects, camera positions, and system configurations.
  We focused our measurements on the horizontal x–y plane only, since this is where the tolerance is most important for a top-down grasp.
  For setup, we removed the robot's gripper, pointed the wrist downward, and hung a weight from a string attached to the middle of the wrist.
  Gravity held the weight directly below the wrist, allowing manual measurement of the lateral distance in the horizontal plane between the center of the wrist and the center of the object. 
  If the system were perfect, the dangling weight would align laterally with the center of the top face of the object.
  
  
  
  
  For this first set of experiments, we explored two design choices using the sugar box from the YCB objects <cit.> (yellow outline in Fig. <ref>).
  The object was placed in four locations around the table, in an approximately 20 cm by 20 cm area, depending upon the camera's field of view and the robot's reach. 
  The object was also placed with an in-plane rotation of approximately 45^∘ to ensure that the camera could see two faces of the object.
  Before manually taking a measurement, we waited several seconds to let the system stabilize, to reduce the effects of the exponential filtering.
  Mean error at these locations was calculated for each configuration.
  
  
  Tab. <ref> shows results of this experiment.
  The first column (“network”) details two design choices of the robot-to-camera network: whether the output dimensions are 100×100 or 400×400, and whether joints (J) and/or torso (T) keypoints were used.
  The dimensions of the object-to-camera network output were fixed to 50×50.
  The second column (“ref”) indicates whether the pose refinement was used.
  Error is measured at a range of camera-to-robot distances (d_i) from 102 to 162 cm, d_i=92+10i cm.
  At these distances, the width of the sugar box in the 640×480 image varies from approximately 99 to 51 pixels.
  Due to space limitations, only mean error is shown.
  
  From the first row, we see that the baseline DREAM network from <cit.>, which was trained only on the robot joints, does not have sufficient information to estimate the camera-to-robot pose at distances closer than 152 cm; and when the camera is far, a sufficient number of joints are seen, but the accuracy is poor.
  The next four rows show the effects of image resolution and pose refinement on our newly trained versions, both of which help to improve accuracy.
  Interestingly, the penultimate row shows that higher-resolution output with fewer keypoints (400×400,T) yields similar results to 
  smaller-resolution output with more keypoints (100×100,JT). 
  Note that with both joints and torso keypoints (JT), performance is more consistent across distances:  closeup, the torso keypoints are visible, whereas farther away, both joint and torso keypoints are visible, which helps to mitigate errors which otherwise would be expected from increasing distance.
  
  
  
  
  
  In the next experiment, shown in Tab. <ref>, we evaluated the performance of the system for 5 YCB objects.
  Objects were placed (as before) at a favorable angle to allow the perception system to view multiple sides.
  For this experiment we used the 100×100, JT network and included pose refinement.
  For most objects, the errors are within the threshold required for grasping at most distances.
  The low accuracy of the mustard was likely caused by the lack of texture on the object, which caused the pose refinement to inaccurately estimate the object to be further away from the camera.
  
  
  
  
  In the previous experiments, we used only a favorable object orientation while varying camera distance.
  In the next experiment, we maintained a constant distance (d=112 cm) and object position while varying the pose of each object at roughly 10^∘ increments in an upright position, along with a couple other orientations (side, upside down), leading to 20–25 measurements per object.
  As before, we tabulated the lateral distance between the object centroid and end effector using the hanging weight.
  Tab. <ref> presents the mean and standard deviation of the readings for YCB objects of three representative shapes (cuboid, cylinder, and bottle).
  We maintained the same system configuration as the previous experiment (last row in Tab. <ref>).
  Other than the mustard bottle, which again due to lack of texture yielded worse performance compared to the other objects, accuracy (even including one standard deviation) was mostly within the grasping threshold.
  
  
  
  
  Finally we measured positioning accuracy on all 28 proposed new objects from a fixed camera distance (d=112 cm).
  We used the same networks as above, but did not use a refinement step, to better directly understand the accuracy of the combined neural networks.
  Tab. <ref> presents the results of placing each object in four distinct locations in front of the robot, as before, with favorable orientation.
  The table is organized by object shape (cuboids, cylinders, and bottles).
  From the results, cuboids are the most reliable, if the spaghetti box is ignored; this object is uniquely challenging due its long and thin shape. 
  Some of the cylinders are also quite difficult due to their small size, and the bottles present similar challenges as the YCB mustard bottle due to lack of texture.
  Another issue is what we call pseudo-symmetry, where only a tiny detail (such as a lip on a box) differentiates two symmetric configurations, the front and back of the cookie box.
  From these results, we believe that these objects will be useful to the robotics and computer vision communities (given that some objects are well detected and stable), as well as challenging for future research (since many objects yield poor accuracy, with errors above 2 cm).
  
  
  
  
  
  
  
   §.§ Camera motion
  
  
  In this experiment we investigate the stability of the system when the camera is moved freely. 
  The sugar box was placed on a table in front of the robot with the robot arm center above it.
  The prior network configuration was used with pose refinement.
  The camera was then manually moved around the room while ensuring that the robot and object remained in the field of view.
  The first row of Fig. <ref> shows the camera movement as perceived by our 
  pose estimator, using the standard coordinate frames of the Baxter robot, the x axis (red curve) points toward the front of the robot, the y axis (green curve) points left (which is approximately the camera's right), and the z axis (blue curve) points up.
  By this estimate, the camera was moved from a depth of 1.3 to 2.0 m and laterally from -0.3 to 0.3 m while its height ranged from 0.3 to 0.5 m.
  
  The second row of Fig. <ref> shows the estimated object position relative to its starting position in the robot coordinate system; since the sugar box did not move, in a perfect system all colored curves should be horizontal lines at zero. 
  With exponential filtering, the estimated position remains largely within or near the 2 cm range of graspability for most of the sequence.
  Because the sugar box was aligned to be parallel to the face plate of the robot, the x direction (red curve) is most sensitive for grasping, the y direction (green curve) is less sensitive, and the z direction (blue curve) is the least important (since it is the approach direction).
  
  The last row of the figure shows end effector displacement relative to its initial position as the closed-loop system attempted to hover over the object.
  As before, in a perfect system all lines would be horizontal at zero.
  The dynamics of the physical system dampens the errors slightly but also introduces slight errors of its own.
  
  Fig. <ref> shows the cumulative distribution function for all three axes of object displacement from this experiment.
  (The plot for end effector displacement is similar.)
  Again, the most important axes are the red and green, in that order, since these are the directions in which the gripper has the least tolerance for error. 
  Despite the unstructured movement of the camera and the use of non-full image resolution, the robot remains within the graspable region approximately 60% of the time.
  In other words, roughly 60% of the time, the robot would have successfully grasped the object had it been commanded to move down and close the fingers.
  
  
  
  
  
  
  
  
  
  
   §.§ Grasping experiments
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  The ultimate test of the presented method is whether its accuracy is sufficient for robotic grasping.
  As such, we designed a simple controller to move the robot gripper above the detected object, 
  descend on it, and close the gripper; if the grasp was successful, the robot then picked up the object and dropped it in a basket. 
  For each object we performed grasps at random locations in the workspace, including while stacked on other objects. 
  Specifically, we performed 5 grasps at 2 different camera locations, yielding 10 grasps per object. 
  Both networks ran continuously during the manipulation; thus, the camera calibration was never fixed.
  For this experiment we used the same network parameters as before. 
  
  Table <ref> shows results using pose refinement for various YCB objects and without refinement for the proposed toy objects. 
  Since Baxter's parallel jaw gripper has a rather small tolerance of 2 cm when grasping an object (as mentioned earlier), we believe these results show the promise of such an approach in real robotics applications.
  Fig. <ref> shows the robot successfully grasping an object using the proposed system; note that the object was continually detected, even while held by the gripper, which could be used in the future to detect slippage.
  
  
  
  
  
  § DISCUSSION & CONCLUSION
  
  
  In this work we have presented a system that computes object poses with respect to the robot indirectly, by combining the output of a neural network that estimates the object-to-camera pose with the output of another network that estimates the robot-to-camera pose.
  The system uses a single external RGB camera as input, which can be moved freely during operation without significantly affecting accuracy, due to the online calibration provided by the latter network.
  We have shown that our system is accurate enough to perform robotic grasps with 2 cm tolerance.
  We also introduce a new set of toy grocery objects that are easily accessible to other researchers due to their availability online.
  To aid reproducibility of our results, we share the 3D scanned textured models and pretrained weights for pose estimation of these novel objects.
  
  It is difficult to convey in a paper the tremendous freedom that online calibration provides.
  As researchers, in the past we have always been very careful not to accidentally nudge a camera after calibration, lest the system break, and a tedious recalibration step be required.
  With this system, however, as we were running experiments, we were able to freely grab the camera and move it around, without fear that the system would stop working.
  To be fair, the exponential filtering is critical for safe performance; but with this filtering we can pick up and move the object or camera, walk around the robot, occlude the camera, and so forth, all while the robot is servoing on the object position.
  In this respect, the performance we observe is remarkably stable.
  
  Having said that, the problem is far from being solved.
  Many aspects of the system can cause grasp failure:  incorrect size of the 3D scanned textured model, inaccurate pose estimation from either network due to occlusion or lighting conditions, pose refinement getting stuck in a local minimum, the object being aligned with the camera image plane so that only one face is visible (in the case of a cuboid, which increases error), and so forth.
  Moreover, for object detection, false negatives occasionally prevent the object's pose from being estimated, which we noticed particularly as the lighting conditions of the room changed from morning to evening; rarely, false positives can cause the robot to wander to strange places in the scene (though this is largely mitigated by the exponential filtering).
  
  For future work, there are many ways to improve the robustness of the proposed system, including using higher-resolution input images and output belief maps, incorporating tracking into the state estimation, fine-tuning the networks with real data, and improving the quality of the 3D scanned textured models used.
  Moreover, achieving reliable grasping with a single external RGB camera is inherently difficult due to the lack of depth information.
  As a result, some combination of depth, stereo, and/or camera-in-hand may be necessary to achieve mission-critical levels of robustness.
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
   
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  § ACKNOWLEDGMENTS
  
  
  The authors are indebted to Timothy Lee, Thang To, Jia Cheng, Jeffrey Smith, Christopher Paxton, and Clemens Eppner for their help. 
  
  
  
  
  
  
  
  
  IEEEtran
  
  
  
  -12cm   
                                    
                                    
                                    
                                    
                                    
  
  
  
  
  