Deep Gated Multi-modal Learning: In-hand Object Pose Changes Estimation using Tactile and Image Data
Tomoki Anzai Kuniyuki Takahashi


Novel Reconciliation Protocol Based on Spinal Code for Continuous-variable Quantum Key Distribution 



     	Xuan Wen Qiong Li Haokun Mao Yi Luo Bingze Yan 


    Received: date / Accepted: date
=======================================================================================================

empty




For in-hand manipulation, estimation of the object pose inside the hand is one of the important functions to manipulate objects to the target pose.
Since in-hand manipulation tends to cause occlusions by the hand or the object itself, image information only is not sufficient for in-hand object pose estimation.
Multiple modalities can be used in this case, the advantage is that other modalities can compensate for occlusion, noise, and sensor malfunctions. 
Even though deciding the utilization rate of a modality (referred to as reliability value) corresponding to the situations is important, the manual design of such models is difficult, especially for various situations.
In this paper, we propose deep gated multi-modal learning, which self-determines the reliability value of each modality through end-to-end deep learning.
For the experiments, an RGB camera and a GelSight tactile sensor were attached to the parallel gripper of the Sawyer robot, and the object pose changes were estimated during grasping.
A total of 15 objects were used in the experiments.
In the proposed model, the reliability values of the modalities were determined according to the noise level and failure of each modality, and it was confirmed that the pose change was estimated even for unknown objects.
[An accompanying video is available at the following link:
 <https://youtu.be/2OPQxQIcSxY>]






§ INTRODUCTION


Robots are expected to not only work in factories, but also in home situations where they are expected to grasp objects in various environments.
Reaching the target grasp pose in a direct motion is difficult when there are obstacles in the environment.
In such situations, the robots need to grasp the object once and then place it on a surface such as a desk to grasp it again, in order to achieve the target pose.
Another way is to rearrange the object inside the hand, also known as in-hand manipulation.
Though commonly seen as the easier solution, placing and re-grasping an object generally takes more time and steps compared to in-hand manipulation.

Furthermore, surfaces to place the object on do not always exist in environments, therefore this method can not always be used.

For robot manipulation tasks, particularly in-hand manipulation, object pose estimation within a robotic hand is one of the important functions to manipulate objects to the target pose accurately <cit.>.

Although many researchers have developed object pose estimation based on image information only, images alone are not sufficient for in-hand manipulation, where the object is likely occluded from the camera by the object or the hand itself (e.g. the gripper hides the object).
Furthermore, large objects can go outside the field of vision of the camera or hide parts of themselves as depicted in Fig. <ref>.
One of the approaches to address these challenges is to combine multiple sensor modalities.
One of the advantages of using multiple modalities is that other modalities can compensate and provide information from different perspectives, even in the case of occlusions, noise, or malfunctions in some of the other modalities.
In such situations, it is necessary to predict how much each modality should be considered.
For the remainder of this article, we refer to the utilization ratio of each modality as its reliability value.
However, it is difficult to manually design a model that can decide a given modality's reliability value, especially if the model has to deal with a wide variety of environments and situations.



In this paper, we propose a method that we call deep gated multi-modal learning (DGML), which uses end-to-end deep learning to predict and determine the reliability value of each modality by DGML itself (See Fig. <ref> and Fig. <ref>).
By virtue of end-to-end deep learning, this method is capable of generalizing to unknown objects without assuming a known 3D model when doing in-hand pose estimation.

The rest of this paper is organized as follows.
The contribution is explained in <Ref> and works related to this paper are described in <Ref>, while our proposed method is detailed in  <Ref>.
<Ref> outlines our experiment setup, and evaluation settings with the results are presented in <Ref>.
Finally, future work and conclusions are explained in <Ref>.





§ CONTRIBUTIONS


The target of our method was to estimate the object pose changes inside the hand using image and tactile sensors as robustly as possible, despite occlusions, noise, sensor malfunctions and other possible obstructions.
Note that we track changes (plural) since we estimate a time-series of how the object pose evolves per time step.
What should also be noted is that our method estimates in-hand relative object pose changes during grasping instead of the absolute object pose with respect to the robot base, meaning that we estimate how much the object moves within the hand after the robot first makes contact with the object (when it has grasped).
This new method is able to dynamically determine the reliability value and scale each modality's contribution appropriately instead of focusing on only one modality or the other.

The main contributions of this article are as follows:

    
  * Proposed a new approach in which the network itself determines the reliability of each modality dynamically and uses that reliability value to scale each modality's contribution.
    
  * Proposed a new approach in which end-to-end learning combines image and tactile data without assuming a 3D model to estimate the pose changes of unseen objects which is robust against occlusions, noise, and sensor malfunctions.
    
  * Investigated details of noise, malfunction, and occlusion behavior in sensor information unique to the robot field.









§ RELATED WORKS




 §.§ Object Pose Estimation with Depth and Image Data


Object pose estimation is a well-studied problem in computer vision and is important for robotic tasks.
Many researchers have particularly been developing methods using depth data (point cloud) or RGB-D data <cit.>.
Classical approaches with depth data are mainly based on point cloud matching methods such as iterative closest point (ICP) <cit.>.
These methods can achieve high accuracy, but since these methods require 3D models of the objects in advance, they cannot be used for unknown objects.

Recently, deep learning has become an active research area and especially the computer vision field has achieved success. 
Image based object pose estimation methods through combination of deep learning with model-based approaches have been studied <cit.>.
The convergence to the final result of ICP heavily relies on the choice of the initial position, but the convergence error can be suppressed by giving a proper initial position through deep-learning methods.
However, these methods still require a 3D model, thus adapting to unknown objects remains challenging.
Some state-of-the-arts pose estimation methods that do not require 3D models have been realized through deep learning <cit.>.



 §.§ Tactile-based Object Pose Estimation


As described in <Ref>, most of the existing pose estimation methods use images and depth information only.
During in-hand object manipulation, the object is occluded from the camera and depth sensor by the hand or the object itself (See Fig. <ref> as examples of occlusions).

Tactile sensors are gaining attention since they can observe the contact state even with occlusions <cit.>.
The majority of these sensors fall in either of the following two categories:

    
  * Single-axis multi-touch sensors, which can only sense normal force <cit.>
    
  * Three-axis single-touch sensors <cit.>

Two of the few exceptions are the uSkin <cit.> and the GelSight <cit.> which are multi-touch sensors that can measure shear force as well as normal force.
The commercialized uSkin sensor <cit.> utilizes embedded magnets inside a silicone rubber and measures the deformation of the silicone during contact by monitoring changes in the magnetic fields.
Using this method, it is able to measure both normal as well as shear force per sensor unit. 
The prototype supports 16 different points of contact <cit.>.
Instead of magnets, GelSight <cit.> is an optical-based tactile sensor, which uses a camera to capture and measure the deformation of the attached elastomer during contact with a surface.

There are some works using tactile sensors for pose estimation such as <cit.>, but a 3D model of the object is required a priori.
It is challenging to estimate the pose of an object using only the tactile information of the grasped part since there is little information from only a single grasp.
Therefore, some work has been done in which tactile and image information are combined <cit.>, and the effectiveness of this fusion was demonstrated in object grasping tasks.
Some of the works performed object pose estimation through model based approaches using known 3D models <cit.>.
In these studies, a 3D model of the object is often required to estimate absolute object pose because it is assumed that the pose of the object is unknown when the object is grasped.
However, we expect that even the methods that work on unknown objects described in <ref> can be improved by using additional tactile information for dealing with occlusions.
The object pose can namely be estimated before the robot grasps an object, through methods without a 3D model of the object such as <cit.> earlier mentioned in <ref>.
If we then keep track of the object pose changes using a tactile sensor after the robot first makes contact (when it has grasped the object and thus occludes it), we will be able to track the absolute object pose without using a given 3D model.
In addition, our hypothesis is that tactile sensors improve the object pose estimation accuracy compared to using images only (Details of our experiment results to test this are described in Table <ref> in section <ref>).




 §.§ Attention Based Learning

A variety of attention methods has been studied.
Regarding the division of attention to each modality, the majority of these methods fall in either of the following three categories, except for our proposed method:

1) Equal attention to all modalities:
Each modality is fed into their respective networks to extract features, and the obtained features are simply combined to estimate information such as grasping point and motion.
For example, the combination of image and tactile data described in <Ref>, motion and image <cit.>, force and image <cit.>, language and image <cit.>, and, image and sound <cit.>.

2) Attention within modalities:
Only the important parts of the given modalities are used.
Within each modality, the part to focus on is extracted by the network itself.
For example, in the case of images, the pixel of interest is used instead of the whole image <cit.>.
This case does not necessarily use multiple modalities as it can also be done when using a single modality.

3) Cancellation of irrelevant modalities:
Only the important modalities are used while other modalities are ignored.
The network decides which modality should be used from all modalities.
For example, choosing to utilize language or images according to the situation <cit.>.

The new approach we developed does not fit in these categories, since it determines the contribution ratio of each modality dynamically and uses that reliability value to scale each modality's contribution.





§ DEEP GATED MULTI-MODAL LEARNING


We propose a method that we call deep gated multi-modal learning (DGML) for in-hand object pose changes estimation with image and tactile information based on end-to-end deep learning.
The DGML estimates a time-series of relative in-hand object pose changes during grasping instead of the absolute object pose.
Figure <ref> shows the concept of the proposed network model.
The main concept of DGML is that the network itself dynamically decides how much each modality it should be relied on, in other words to decide the reliability value per modality.
We aimed to design a network with a structure that is as simple as possible, but still sufficient to show the effectiveness of the deep gated multi-modal unit.
Increased complexity of the network architecture will most likely improve the accuracy, but the concept of reliability values can be used in the same way.
The details of the network structure will be described in section <ref>.

DGML is composed of three components for in-hand pose estimation:

    
  * Feature extraction unit to extract features from image and tactile data
    
  * Deep gated multi-modal unit for calculation and application of a reliability value for each modality
    
  * Object pose changes estimation unit to estimate object pose changes using time-series input information

For training and inference of the network, a sequence of image and tactile data are the input, and the output is a sequence of object pose changes.
A training dataset D = {(input_1,output_1),...,(input_T,output_T)}_n consists of n sequence with T steps. 

As a feature extraction unit, convolutional neural networks (CNNs) are used to calculate image features x_img(t) and tactile features x_tac(t) at step t from image input I_img(t) and tactile input I_tac(t), respectively.

Then, the reliability value α for image in step t is given from Gate as:

    α(t)     =    Gate(x_img(t), x_tac(t)) 
       =    Sigmoid(FC_img(x_img(t)) + 
                           FC_tac(x_tac(t)))

where FC_img and FC_tac are fully connected (FC) layers to reduce the number of dimensions, and Sigmoid is used as activation function for the gate.
The reliability values α(t) for image and β(t) for tactile data are conditioned to sum up to 1:

    α(t) + β(t) = 1

α(t) and β(t) are one-dimensional scalars.

The gate determines each modality's reliability values from all modalities' information through equations (<ref>) and (<ref>).

The extracted feature vectors for each modality, x_img(t) and x_tac(t), are then scaled by multiplying them with their respective reliability values, giving us the scaled feature vectors h_img(t) and h_tac(t) for time step t:

    h_img(t)    =   α(t) · x_img(t) 
    
        h_tac(t)    =   β(t) · x_tac(t)

Since a sigmoid function is used and the total sum of reliability values is 1, the reliability values are continuous values from 0 to 1, instead of being only 0 or 1.
The lower the reliability, the smaller the contribution to the output.
On the other hand, the greater the reliability, the greater the output contribution.
In addition, if the reliability value becomes 0, the modality is completely ignored.
Therefore, it is not necessary to manually determine which sensor to enable/disable in advance, because the network will automatically ignore them if they are not helpful.
Note that the reliability values are not absolute, but relative.
We chose to use relative values to learn the correlation between multiple modalities.
This however means that the method simply assigns a higher reliability value to a given modality, if said modality performs better than any other modality.
Thus, all modalities could be noisy or under-performing, but our method will still assign reliability values by comparing the modalities' performances with each other.
The downside of this is that the robot will continue to operate even if none of the sensor modalities are performing well.
Rather, it tries to make the best out of the situation and attempts to focus on only the good modalities of the data it has.
Detecting when the robot should give up trying or improving the quality of the data however is out of scope of our work.

As the object pose changes estimation unit, long short-term memory (LSTM) with its output connected to FC is used.
Input of the LSTM is a sequence of h_img(t) and h_tac(t), whereas the output of FC is O^'(t)=Δp^'(t) ∈R^d, which is the estimated sequence of object pose changes from the LSTM.

Since our proposed method works without 3D models given in advance, we estimate the relative object pose (object pose changes) during grasping instead of the absolute pose.
Thus, object pose change Δp(t) is the current object pose at step t with respect to the initial object pose at step 1 when the robot first grasps the object and makes contact.
When t=1,  O(1)= Δp(1) = 0.









The loss function L is minimized as follows:

    min_ξ1/mT∑_i=1^m∑_t=1^T L(O(t), O^'(t))

where ξ are the parameters to be trained, m(≦ n) is the number of sequences for mini-batch training, and O(t) ∈R^d is the teaching signal.


There is no teaching signal for the reliability values α and β because these are calculated by the network itself to minimize the output error by equation (<ref>).
Then, the reliability values are multiplied by each modality in equation (<ref>).






§ EXPERIMENTAL SETUP


The purpose of the experiments is to verify DGML in situations with occlusion, noise, and sensor malfunctions.

We note that our values of the hyper-parameters provided in this section are tuned by random search.



 §.§ Hardware Setup



  §.§.§ Tactile sensor

The GelSight tactile sensor we duplicate from article <cit.> is an optical-based tactile sensor, which captures a 640×480 image by a camera, which can then be used to calculate the 3D model and the applied normal and shear force in the x, y, and z axes in Newton.
However, instead of using the 3D model and the force in Newton we directly use the captured raw image from the GelSight (See Fig. <ref>).
Because the silicone layer from the sensor's container tears when an excess amount of force is applied, we applied baby powder on this layer to reduce friction between the surface and the grasped object.
The reason why we chose GelSight is that it is a multi-touch sensor that can measure both normal and shear forces as opposed to other types of tactile sensors described in section <ref>.
It is also relatively cheap and simple to reproduce.







  §.§.§ Gripper

We developed a gripper to grasp objects shown in Fig. <ref>.
This gripper is a parallel gripper which has two fingers driven by a servo motor (Dynamixel XM430-W350-R).
A GelSight tactile sensor is attached to one fingertip, and the other fingertip has a sponge.
A camera (BUFFALO BSW200MBK) is mounted at the center of the gripper.




  §.§.§ Sawyer

To perform our experiments, we use a Sawyer 7-DOF robotic arm with the gripper as end-effector (See Fig. <ref>).
The Sawyer, GelSight sensor, gripper, and camera are connected to a PC running Ubuntu 16.04 with ROS Kinetic.





 §.§ Objects

For the target objects, we have prepared 15 objects with various size and shape (See Fig. <ref>).
11 of these objects are used for training, while the remaining 4 were used to evaluate our trained network as unknown objects.



 §.§ Data Collection


Figure <ref> shows one of the initial positions of the robot from which it starts to manipulate the object for data collection (The attached video shows more examples of the initial positions).
A table is placed in front of the robot and the object is fixed on the table with double-sided tape.
After the robot grasps the object, the gripper posture, image, and tactile data are recorded while the robot slides the object in its hand.
Since we use a parallel gripper popular in robotics, the object pose is limited to translation in the xy plane and rotating θ around the z axis.
We estimate the three DoF object pose changes given the coordinate system of the hand, which is depicted in Fig. <ref>.

The ground truth object pose change Δp(t) is calculated from homogeneous transformation matrix ^O_1_O_tH = ^R_O_1H^-1^R_O_tH, where
^R_O_tH is the pose of the object at time t with respect to the robot. ^R_O_1H is the pose of the object when the robot first grasps the object and makes contact.
The pose of the object is the inverse transformation of the posture of the gripper, since the object is stationary and only the gripper moves along the object. So ^R_O_tH = ^R_G_tH^-1; this can be calculated easily through forward kinematics.
 
Since we define object poses with respect to the gripper pose at initial grasp contact, what we really estimate are the relative object pose changes between the pose at the current step t and initial step 1 when the robot first makes contact with the object (when it has grasped).
Thus, the object pose change is not the absolute pose of the object.
However, the absolute pose can be calculated if the robot can estimate the absolute object pose before grasping as described in section <ref>.
Thus, the teaching signal and ground truth O(t) are defined as follows:

    O(t)    =   Δp(t)=(Δx(t), Δy(t), Δθ(t))


We decided to collect the teaching signals/ground truth though means of forward kinematics and by fixing the object to the table because this results in a higher accuracy as compared to other non-fixed methods such as AR marker tracking.
We also prepared the movement patterns including both translational and rotational motions, and collect data for each object (The attached video shows examples of the motions).
In order to prevent learning features from the background as the robot moves, we covered the background and desk in green clothes.
The maximum movement in translation was about 30 mm and for rotation the maximum was about 40 degrees.
For small objects like tapes, cups, scale, and wrench, only rotational movement was performed (See Fig. <ref>).
For each object, the number of motions for translation, rotation, and a combination of both of them are 10, 10, and 12, respectively. 
Each grasp posture is different in each motion.
Note that even if the gripper does not occlude the object, the object can still be out of view of the camera if the object is large, and thus “occlude“ itself.
For the trained objects in Fig. <ref>, 6 out of 10 translations, 6 out of 10 rotations, and 8 out of 12 combined motions are used for training, and the remaining sets are used for evaluation.

Images, tactile, and object poses were acquired at 30 Hz, and the dataset used for training was re-sampled to 15 Hz.

The captured images were converted to gray-scale because the object pose is independent of the color of the objects.
One motion of the training dataset has a length of about 150 steps, and each step is composed of an 80 × 120 pixels image, a 160 × 120 pixels tactile image, and a 3 DoF object pose change, (Δx(t), Δy(t), Δθ(t)).




 §.§ Network Design


The architecture of our network model is composed of two CNNs, gate, and LSTM with FC to perform DGML as shown in Fig. <ref> and as described in <Ref>.
We used Chainer <cit.> as deep learning library for implementation.
More details on the network parameters are shown in Table <ref>.
For training, we used the Huber loss as loss function L.
Due to computer resources, we reset the history of LSTM every 20 steps.
All our network experiments were conducted on a machine equipped with 256 GB RAM, an Intel Xeon E5-2667v4 CPU, and eight Tesla P100-PCIE with 12GB resulting in about 24 to 48 hours of training time.









§ RESULTS




 §.§ Learning Curve of DGML

As a comparison with DGML, we prepared several networks that use only images, only tactile data, and both images and tactile data with a simple connection, which is without using a gate.
The number of layers and parameters of CNN and LSTM are the same in all comparison models.
However, if one modality is disconnected, the number of input neurons of LSTM changes, since the number of input neurons is the sum of the number of neurons connected per modality.
In addition, a simple connection without using a gate is the same as DGML with fixed reliability values α(t)=1 & β(t)=1.

Figure <ref> shows the learning curves and average reliability values of all sequences in DGML.
Note that the reliability values of α(t) and β(t) are not constants but change dynamically depending on the input I_img(t) & I_tac(t).
As for the reliability values of α(t) for image and β(t) for tactile, α(t) is greater than  β(t) at first, but it can be seen that the reliability value for tactile features gradually increases.
This means that the network gradually relies more on tactile data as opposed to images, which is most likely because the images become less reliable due to occlusions.
As one of the characteristics of DGML, the computational epoch until convergence is the fastest of all models since training progresses from easy-to-train modalities.

Note that the size for I_tac is twice the size of that of I_img, but this is merely a result of the parameter search we performed, and it gave us the best performance of the network used in this study.
As a result, the size of x_tac(t) is twice as large as x_img(t) for the same reason.
The reliability values of α(t) and β(t) can change depending on these parameters.
However, the focus of this work is to improve the performance by introducing DGML with reliability values, so comparing the accuracy when the same input size per modality is used is out of the scope of this work.








 §.§ Inference Result of Object Pose


Table <ref> shows the object pose changes inference accuracy during in-hand manipulation.
The accuracy for translation and rotation is the average difference between the ground truth (Δx(t), Δy(t), Δθ(t)) and inferred object pose change (Δx^'(t), Δy^'(t), Δθ^'(t)) at each time step and is calculated according to equation (<ref>):

    acc_trans   =   1/nT∑_i=1^n∑_t=1^T (|Δx(t)-Δx^'(t)| + 
                      |Δy(t)-Δy^'(t)|) 
    
        acc_rot   =   1/nT∑_i=1^n∑_t=1^T (|Δθ(t)-Δθ^'(t)|)

The ground truth is the object pose change as calculated during the data collection (see section <ref> and equation (<ref>)). 
We evaluated the models under normal conditions, but also by substituting either image or tactile input with random noise.
Under these conditions, we compared four different models: a model using only image input, a model using only tactile input, a model using both image and tactile (but no gate), and the proposed model.

From the results under normal conditions, the results are always better when tactile information is included as we already expected in section <ref>.
In addition, the models including tactile information can predict correctly for both known objects and unknown objects.

From the results with one of the modalities muted in Table <ref>, we can see that the performance of the proposed DGML is the best.
When the image input is absent, the α value is much smaller than the β value (See Table <ref>).
On the other hand, in the case of absence of tactile input, the β value becomes almost 0.
This is because the gate decides that the reliability value of a modality should be reduced if that modality's input is absent, resulting in near ignorance of said modality (See Table <ref>).
Even though the training dataset does not include data where a modality is completely absent, the gate of DGML can still deal with these situations.



 §.§ Gate Values under Noise

In this section, we discuss the change of reliability values when noise is applied to the input of a modality.
The noise is applied to tactile input since in-hand pose changes estimation relies more on tactile information than image information according to section <ref>.
Figure <ref> shows a histogram of the reliability value α for images when the network infers from the original dataset, but with noise added to the tactile input I_tac(t).
The noise is generated from a normal distribution with different variances σ^2: 400, 450, and 500 (See Fig. <ref>).

From Fig. <ref>, it can be seen that the reliability value for images increases as the noise to the tactile input increases.
The gate can correctly recognize the noise applied to the tactile input data.
The strength of using DGML is that DGML helps to understand the network behavior through the reliability values of the gate because the gate represents how much each modality should be used.
Furthermore, if a sensor is broken, the reliability value for that sensor is always close to 0, thus DGML can recognize sensor failures.








 §.§ Reliability Values of Objects

In this section, we discuss the change of the reliability values of objects due to object shape, size, and occlusions.
We visualize the reliability value of images, α, by creating a histogram with all the inferred values of α through untrained motions with both known and unknown objects (See Fig. <ref>).
Most of the reliability values of α are smaller than 0.5, which indicates that the network relies more on the tactile features in those cases.
From the comparison between only using images and only using tactile data in section <ref>, we draw the hypothesis that the reliability values for images are low because the method that uses only tactile data has higher accuracy than the one using only images (See Table <ref>), which seems to indicate that tactile data is more useful for this task.

The objects are depicted on the most frequently occurring α on the histogram in Fig. <ref>.
The ratio between the object sizes in the image and the actual object sizes is the same.
Since objects with α values between 0 and 0.2 such as tapes, wrenches, and cups tend to provide rich tactile information due to their unique shape and/or surface, the network relies more on tactile information for these objects.
Conversely, image information is reliable for objects with no surface irregularities and are displayed with α values between 0.4 and 0.6 such as boxes and cans, for which tactile features are difficult to observe from.
Although some objects have no tactile features on the surface, the reliability of the image is low when such an object is large and thus occludes itself or doesn't fit within the camera view, as opposed to small objects (See top row of Fig. <ref>).
For example, the wood, the plastic coffee bottle, and the orange triangle are small, thus their movements can be observed from images.
In contrast, large objects such as the mustard and green plastic bottles create occlusions, which reduces the reliability of image information, therefore the network uses more tactile information.
Therefore, we can say that the proposed method is effective in determining the reliability value of each modality according to the object shape, size, and occlusions.





§ CONCLUSION


In this paper, we proposed a method for in-hand object pose changes estimation using image and tactile data, while also predicting the modalities' reliability values, called deep gated multi-modal learning (DGML).
The proposed method can estimate not only known object pose changes but also unknown object pose changes during grasping, since it doesn't need 3D models.
Moreover, the modalities' reliability values can be changed dynamically and automatically by the network depending on situations such as sensor failure and different magnitudes of noise.
Visualization of reliability values helps to understand the network behavior such as which modality should be utilized and how much.
Furthermore, by using the proposed method, computation efficiency for training has been improved.

For future work, we will develop a system of in-hand object manipulation to integrate the object pose estimator.





§ ACKNOWLEDGMENT
 The authors would like to thank Wilson Ko for helping to discuss and writting, and Koichi Nishiwaki and Tianyi Ko for proofreading.


IEEEtran 

