PREPRINT SUBMITTED TO ELSEVIER

1

Robotic Tactile Perception of Object Properties: A
Review

arXiv:1711.03810v1 [cs.RO] 10 Nov 2017

Shan Luo* , Joao Bimbo, Ravinder Dahiya and Hongbin Liu

Abstract—Touch sensing can help robots understand their surrounding environment, and in particular the objects they interact
with. To this end, roboticists have, in the last few decades,
developed several tactile sensing solutions, extensively reported
in the literature. Research into interpreting the conveyed tactile
information has also started to attract increasing attention in
recent years. However, a comprehensive study on this topic is yet
to be reported. In an effort to collect and summarize the major
scientific achievements in the area, this survey extensively reviews
current trends in robot tactile perception of object properties.
Available tactile sensing technologies are briefly presented before
an extensive review on tactile recognition of object properties.
The object properties that are targeted by this review are shape,
surface material and object pose. The role of touch sensing
in combination with other sensing sources is also discussed. In
this review, open issues are identified and future directions for
applying tactile sensing in different tasks are suggested.
Index Terms—Tactile sensing, robot tactile systems, object recognition, sensor fusion, survey.

I. I NTRODUCTION

T

HE sense of touch is an irreplaceable source of information for humans while exploring the environment in
their close vicinity. It conveys diverse sensory information,
such as pressure, vibration, pain and temperature, to the
central nervous system, assisting humans in perceiving their
surroundings and avoiding potential injuries [1]. Research
has shown that, compared to vision and audition, the human
sense of touch is superior at processing material characteristics
and detailed shapes of objects [1], [2]. As for humans, it is
essential that robots are also equipped with advanced touch
sensing in order to be aware of their surroundings, keep away
from potentially destructive effects and provide information
for subsequent tasks such as in-hand manipulation.
A general block diagram of a tactile sensing system [3], [4]
is illustrated in Fig. 1. A tactile sensing system here refers
to a system where a robot uses tactile sensors to sense the
ambient stimuli through touch, acquiring information on the
properties of objects, such as shape and material, providing
action related information, such as object localization and
*

Corresponding author.
S. Luo and H. Liu are with the Centre for Robotics Research, Department
of Informatics, King’s College London, London WC2R 2LS, U.K. (e-mail:
shan.luo, hongbin.liu@kcl.ac.uk). S. Luo is also with School of Civil Engineering and School of Computing, University of Leeds, Leeds LS2 9JT, U.K.,
and Brigham and Women’s Hospital, Harvard Medical School, 75 Francis St,
Boston, MA 02115.
J. Bimbo is with the Department of Advanced Robotics, Istituto Italiano di
Tecnologia, 16163 Genova, Italy (e-mail: joao.bimbo@iit.it).
R. S. Dahiya is with the Electronics and Nanoscale Engineering Research
Division, University of Glasgow, Glasgow G12 8QQ, U.K. (e-mail: ravinder.dahiya@glasgow.ac.uk).

Fig. 1: Hierarchical functional (left) and structural (right)
block diagrams of robotic tactile sensing system [3], [4]. The
perception blocks have been highlighted that the paper is
contributing.

slippage detection. On the left side of Fig. 1, the tactile
sensing process is divided into functional blocks that depict
sensing, perception and action at different levels. We follow
existing literature [2], using the term “perception” to refer to
the process of observing object properties through sensing. The
right side of Fig. 1 shows the structural blocks of hardware
that correspond to those functional blocks. The sensing process
transduces the external stimuli (e.g., pressure, vibration and
thermal stimulus), into changes on the sensing elements of the
tactile sensors [3], [4]. This data is acquired, conditioned and
processed using an embedded data processing unit, and then
transferred to the higher perception level to construct a world
model, perceive the properties of interacted objects, (e.g.,
shape and material properties). While perceiving, the sense
of touch may possibly be fused with other sensing modalities
such as vision and auditory perception. Control commands are
ultimately to be exerted in order to obtain the desired actions
by the controller.
The last few decades have witnessed a tremendous progress in
the development of tactile sensors, including diverse materials
and methods explored to develop them, as extensively reported
in [3]–[8]. Research into the interpretation of tactile data to

PREPRINT SUBMITTED TO ELSEVIER

extract the information it conveys, particularly in the task of
object recognition and localization, is nowadays also beginning
to attract increasing attention. A comprehensive discussion on
tactile sensory data interpretation is needed since many commonly used techniques, for example those adapted from the
field of computer vision, may not always be suitable for tactile
data. This is because of the fundamentally different operating
mechanisms of these two important sensory modalities.
To assist the advancement of the research in information extraction from tactile sensory data, this survey reviews the stateof-the-art in tactile perception of object properties, such as
material identification, object recognition and pose estimation.
In addition, work on perception that combines touch sensing
with other sensing modalities, e.g., movement sensors and
vision, are also studied.
The remainder of this paper is organized as follows: Available
tactile sensors are first briefly introduced and compared in
Section II. Research on material recognition via touch is then
presented in Section III. Works on tactile shape recognition
and object pose estimation are reviewed on both local and
global scales in Section IV and V respectively. How vision and
touch have been combined for object perception is discussed
in Section VI. The last section concludes the paper and points
to future directions in interpreting tactile data.
II. TACTILE PERCEPTION AS REPRESENTATION AND
INTERPRETATION OF HAPTIC SENSING SIGNALS

A. Tactile sensing modalities
As introduced in Section I, tactile perception is the process
of interpreting and representing touch sensing information to
observe object properties. In the hierarchy presented in Fig. 1,
it is placed a level above sensing, and provides useful, taskoriented information for planning and control [9]. How tactile
sensing information is interpreted and represented is closely
linked with the type of hardware used and with the task to
be fulfilled by the robot. Tactile sensing has been extensively
reviewed in [3], [4], [10], [11] and the tactile sensors can
be categorized in multiple manners, such as according to
their sensing principles [12], fabrication methods [3] and
transduction principles [4]. In this paper, we follow existing
literature [13], [14] and choose to categorize tactile sensors
according to the body parts they are analogous to. As shown in
Fig. 2, tactile sensors in literature can be categorized into three
types with respect to their spatial resolution, and an analogy
can be made to corresponding parts of the human body. We
provide a short description of each category in order to frame
this review as follows.
• Single-point contact sensors, analogous to single tactile
cells: This kind of sensor is used to confirm the object-sensor
contact and detect force or vibrations at the contact point.
Depending on the sensing modalities, single-point contact
sensors can be categorized into: 1) force sensors for measuring
contact forces, where a typical example is the ATI Nano
17 force-torque sensor; 2) biomimetic whiskers, also known
as dynamic tactile sensors, for measuring vibrations during
contact [15]–[19].

2

Fig. 2: Tactile sensors of different types with the corresponding
biological body parts in anatomy.
• High spatial resolution tactile arrays, analogous to human
fingertips: Most research in tactile sensing is carried out using
this type of tactile sensors [3], [11], and example prototypes
are tactile arrays of 3×4 tactile sensing elements based on
fiber optics [20], tactile array sensors based on MEMS barometers [21] and fingertip sensors based on embedded cameras
[22]–[25]. There are also a multitude of commercial sensors
available, such as RoboTouch and DigiTacts from Pressure
Profile Systems (PPS)1 , tactile sensors from Weiss Robotics2 ,
Tekscan tactile system3 , BioTac multimodal tactile sensors
from SynTouch4 . Among them, the most common are planar
array sensors.
• Large-area tactile sensors, analogous to skin of human
arms, back and other body parts: Unlike in fingertip tactile
sensors, high spatial resolution is not essential in this type
of tactile sensing. More importantly, they should be flexible
enough to be attached to curved body parts of robots. The
attention to developing this type of sensors has emerged in
recent decades [6], [26], [27]. Some researchers have developed this kind of tactile sensors for hands [28], arms and legs
[29], [30], front and back parts [31] of humanoid robots. For a
comprehensive review of large area and flexible tactile skins,
the reader is referred to [3], [8], [32], [33].
B. Tactile Perception
Representations of tactile data are commonly either inspired
by machine vision feature descriptors, where each tactile
element is treated as an image pixel [34], [35], biologically
inspired [36], or resort to dimensionality reduction [37]–
[39]. Tactile information can be interpreted according to the
desired function of the robot. Relevant information that can be
extracted from sensing data include shape, material properties,
and object pose. In this review, we focus on existing methods
to extract these parameters, which are fundamental in the
field of robot grasping and manipulation [40], and have been
1 www.pressureprofile.com/digitacts-sensors
2 www.weiss-robotics.com/en/produkte/tactile-sensing/wts-en/
3 www.tekscan.com/
4 www.syntouchinc.com/

PREPRINT SUBMITTED TO ELSEVIER

used for the purpose of grasp control [41], slippage detection
and prevention [42], grasp stability assessment [43], among
others. Other applications where tactile perception has been
successfully applied range from haptic cues for Minimally
Invasive Surgery (MIS) [44], creating interfaces for interactive
games [45] or medical training simulators [46], and assisting
underwater robotic operations [47].
Compared to the rapid development of tactile sensors, the
interpretation of tactile sensors readings has not yet been
fully taken into consideration. This is reflected by the small
number of survey articles reported in the literature that focus
on reviewing the computational intelligence methods applied
in tactile sensing. The development of tactile sensors of increasing spatial resolution and fast temporal response provides
an opportunity to apply state-of-the-art techniques of machine
intelligence from multiple fields, such as machine learning,
signal processing, computer vision and sensor fusion, in the
field of tactile sensing. Accordingly, this paper explores recent
advances in object shape and material recognition and pose
estimation via tactile sensing and gives guidance towards
possible future directions. In addition, it is also investigated
how vision and touch sensing modalities can be combined for
object recognition and pose estimation.
III. M ATERIAL RECOGNITION BY TACTILE SENSING
The material properties of an object’s surface are one of the
most important cues that a robot requires for the sake of
effectively interacting with its surroundings. Vision has been
a popular approach to recognize object material [48]–[50].
However, vision alone can only recognize a previously known
surface material, and cannot, on its own, estimate its physical
parameters. In this respect, it is essential to utilize the sense of
touch to identify the material properties. Surface texture (friction coefficients and roughness) and compliance are amongst
the most crucial parameters for manipulating objects. Humans
are extremely skilled in recognizing object material properties
based on these cues [51] and a comprehensive review on
human perception of material properties can be found in [52].
In robotics, researchers have endeavoured to enable a robot
to identify the material properties at a level comparable to
humans. These properties can be categorized into two different
methods, i.e., surface texture based and object stiffness based.
A. Surface texture based tactile material recognition
Surface texture information can be extracted through the investigation of friction coefficients, roughness and micro-structure
patterns of objects. The former two can be obtained using a
force or tactile sensor sliding on the object surface, whereas
the latter can be attained through tactile images. The friction
that arises at the contact while the sensor is sliding on an
object’s surface can be used to recognize the surface materials.
Among the methods of this kind, the use of acoustic signals
resulted from the friction to recognize the object materials is
low-cost and requires limited computational power [53]. In
[54], a microphone is mounted on a robot leg which taps on
the ground as the mobile robot moves, similar to the manner a
blind person might tap his cane. The acoustic signature from

3

tapping is then used to classify different floor materials. In
[55], an artificial finger equipped with a microphone (i.e.,
dynamic tactile sensor) is used to collect frictional sound data
that are mapped to frequency domain through Fast Fourier
Transform (FFT) to detect different textures. To recognize
different texture surfaces, in [15] mean Maximum Covariance
Analysis (µMCA) and weakly paired MCA (WMCA) are
used to analyze the acoustic data after Fourier transform,
which were collected by a dynamic tactile sensor. In [56], a
microphone-based texture sensor is employed and the textures
can be classified using Self-Organizing Maps (SOMs). The
use of acoustic signals has merits of low-cost and limited
computational expense, however, ambient and motor noise
may deteriorate the recognition performance.
Strain gauges and force sensors are also used to detect vibrations during object-sensor interaction, in order to discriminate
materials. By transferring raw data into the frequency domain
by FFT, in [57], different materials are classified based on
surface textures by analyzing induced vibration intensities. In
these works, an artificial finger is used with strain gauges and
polyvinylidene fluoride films (PVDFs) embedded in silicone.
In [58], a dynamic friction model is applied to determine
the surface physical properties while a robotic finger slides
along the object surface with varying sliding velocity. The
yarn tension data from a fabric sensor are also collected while
the fingertip slides over objects in [59], with multiple computational intelligence methods for recognition being compared.
In [60], a tactile probe is proposed to measure the vibration
signals while sliding to classify disks with different textures.
This tactile probe is also used in [61] to identify textures
of different terrains. In [62], artificial fingernails with threeaxis accelerometers are used to scratch on surfaces and a
frequency-domain analysis of the vibrations is done by using
machine learning algorithms, i.e., kNN and SVM. Similarly,
vibrations are also used in [63], in line with contact forces, to
classify different surfaces. In addition to vibrations and force
data, the output of proximity sensor is used to distinguish
different surface textures in [64].
Roughness is another important cue to discriminate between
different object materials. In [65], object roughness is computed based on the variances of strain gauge signals using
wavelet analysis. A microelectromechanical systems (MEMS)
based tactile sensor is used to discriminate the roughness
of object surfaces in [66]. Using a BioTac sensor in [67],
Bayesian exploration is proposed to discriminate textures,
which selects the optimal movements adaptively based on
previous experience; good recognition performance is achieved
for a large dataset of 117 textures.
The micro-structure patterns of objects can also be utilized to
recognize object materials, usually with tactile array sensors.
By using the camera-based GelSight sensor [24], height maps
of the pressed surfaces are treated as images to classify
different surface textures using visual texture analysis [68].
Similarly, another camera-based tactile sensor TacTip is also
used to analyze the object textures in [69]. In [70] a MEMS
based tactile array sensor is employed to distinguish simple
textures by using Maximum Likelihood (ML) estimation. The
probability density function (PDF) of each texture type is

PREPRINT SUBMITTED TO ELSEVIER

4

created based on the mean and variance of the obtained tactile
arrays and the textures are estimated by maximizing the PDFs.
In [71], tactile images are also utilized to classify terrains, i.e.,
wood, carpet, clay and grass.
(a)

B. Object stiffness based tactile material recognition
Object stiffness is also one of the critical material properties
[72]. By using a BioTac sensor, the object compliance (the
reciprocal of stiffness) can be estimated either using the
contact angle of the fingertip [73] or investigating BioTac
electrode data [74]–[76]. In [77] a robot leg equipped with an
accelerometer is employed to actively knock on object surfaces
and by analysing the sensor data, the hardness, elasticity
and stiffness of the object can be revealed. In recent works
[78], [79], the hardness of objects can also be estimated
by processing the tactile image sequences from a GelSight
sensor. In [80], multiple computational algorithms are applied
to classify various materials based on mechanical impedances
using tactile data and it is found that SVM performs best. In
[81], by using image moments of tactile readings as features,
dynamic time warping is used to compare the similarity
between time series of signals to classify objects into rigid
and deformable. A force sensor is used to test the mechanical
impedances of materials in terms of shear modulus, locking
stretch and density in [82]. A multi-indenting sensing device
is proposed in [83] to measure the stiffness of the examined
object, i.e., phantom soft tissue used in this work.
In summary, object materials can be recognized by using different touch sensing cues based on surface textures, vibrations
and mechanical impedances, summarized in Table I with a
comparison of their pros and cons.
IV. TACTILE OBJECT SHAPE PERCEPTION
Object shape perception is the ability to identify or reconstruct
the shape of objects. The goals of shape perception vary in
different tasks, from capturing the exact shape, like getting the
point cloud of the object, to classification of shape elements or
overall profiles. This capability is crucial for robots to perform
multiple tasks such as grasping and in-hand manipulation.
The more complete information is obtained about the object’s
shape, the more capable the robot will be to plan and execute
grasping trajectories and manipulation strategies. Research
into shape recognition has been dominated by vision based
methods [86], [87]. However, visual shape features cannot be
observed when vision is occluded by hand or in poor illumination conditions. In contrast, tactile object shape perception
is not affected by such factors and can observe detailed shapes
by sensor-object interactions. The surge of high-performance
tactile sensors gives rise to the emergence and rapid spread of
algorithms to recognize object shapes via touch.
The perception of object shapes can be done on two scales,
i.e., local and global, as illustrated in Fig. 3. The former can
be revealed by a single touch through tactile image analysis. It
is analogous to the human cutaneous sense of touch, which is
localized in the skin. The latter reflects the contribution of both
cutaneous and kinaesthetic feedback, e.g., contours that extend
beyond the fingertip scale. In this case, intrinsic sensors,

(b)

Fig. 3: Two ball shapes with different local shapes: one is a
golf ball and the other is a tennis ball. At a global scale, both
of them are balls. At a local scale, (a) has small pits whereas
(b) has curvilinear shapes.
i.e., proprioceptors in joints, are also utilized to acquire the
position and movement of the fingers/end-effectors that are
integrated with local features to recognize the objects. Here
the kinaesthetic cues are similar to human proprioception that
refers to the awareness of the positions of the body parts.
A. Local shape recognition
In terms of identifying local shapes, recent years saw a trend
to treat pressure patterns as images, thereby extracting features
based on the pressure distribution within the images [35].
The increasing spatial resolution and spatio-temporal response
enable tactile sensors to demonstrate the ability to serve as an
“imaging” device. A large number of researchers have applied
feature descriptors from vision such as image moments [35],
[88] to tactile data to represent local shapes. However, there
are differences between vision and tactile imaging as listed in
Table II. For vision, the field of view (FoV) is large and global
since multiple objects can exist in a single camera image. A
great amount of features can be obtained from one single
image and it is relatively easy to collect data by cameras.
On the other hand, it requires high computational resources
to process the visual data; there are several fluctuation factors
that affect the performance of extracted features, e.g., scaling,
rotation, translation and illumination. Here scaling is caused
by the distance from cameras to observed objects. In contrast,
for tactile sensing, the FoV is small and local as direct sensorobject interactions need to be made. And the information
available in one reading is limited due to the low sensor
resolution (for instance, Weiss tactile sensor of 14×6 sensing
elements compared to a typical webcam of 1280×1024 pixels),
especially in terms of revealing the appearance and geometry
of objects. Compared to vision, it is relatively expensive to
collect tactile data, in terms of sensors and robot components
but there is less effort involved, as it requires less computations
to process the tactile data. In addition to these properties,
features can be extracted from the tactile data that include
surface texture [84], mechanical impedance [73] and local
detailed shapes. In tactile imaging, the influence of scaling
is removed as the real dimension and shape of the interacted
object can be mapped to the tactile sensor directly, whereas
the impact of rotation, translation and “illumination” remains.
Here, “illumination” refers to different impressions of object
shapes caused by forces of various magnitudes and directions,
similar to the variety of light conditions in vision.
1) Shape descriptors for tactile object recognition: As raw
tactile readings are to some extent redundant, features are

PREPRINT SUBMITTED TO ELSEVIER

5

TABLE I: A summary of material recognition methods with touch sensing
Methods

Motions

Data types

Advantages

Disadvantages

Applicable sensors

Sliding,
tapping,
scratching

Frictions, acoustic data, acceleration, vibrations

Low cost and limited
computational expenses

Interaction actions (sliding, tapping, scratching)
may damage objects

Dynamic tactile sensors,
force
sensors,
strain
gauges and PVDF sensors

Imprint

Tactile images

Micro-structure patterns
of object textures can be
captured in one tactile
image

Low resolution of tactile
sensors may cause difficulties for processing

Planar array sensors; potentially applied to curved
tactile sensors

Squeezing,
knocking,
pressing

Force variances,
tactile
images,
acceleration

Limited
computational
costs; complementary to
other information

Interaction
actions
(knocking,
squeezing)
may damage objects

Accelerometer-based sensors, force sensors and
tactile array sensors

Texture based

Mechanical
impedances
based

Ref.
[56],
[62],
[84]
[68],
[70]
[64],
[77],
[85]

TABLE II: Comparison of vision and tactile sensing

• Descriptors adapted from computer vision: In these
methods, tactile arrays are treated as images and thus vision
descriptors can be applied and adapted. Several researchers
Global Rich
Low
Visual
took image moments as feature descriptors [35], [43], [81],
Local
Sparse High
Tactile
[88], [97]. For a tactile reading f (x, y), the image moment
mpq of order p + q can be calculated as follows:
Note: FoV: Field of View; Info.: Information; Compl.: Complexity to collect
XX
data; Compu.: Computation. “illumination”: different impressions caused by
mpq =
xp y q f (x, y),
(1)
forces of various magnitudes and directions.
Modality

FoV

Info.

Compl.

Compu. Invariance
Scaling, rotation, transHigh
lation, illumination
Rotation, translation,
Low
“illumination”

x

utilized to represent the collected tactile data. The paradigm is
to extract features based on the pressure distribution in tactile
arrays and then feed the extracted features into classifiers to
build a training model that can be used to recognize test object
shapes. According to the descriptors used, methods for local
shape recognition can be categorized as follows:
• Raw tactile readings as features: This method avoids the
feature extraction process and is easy to implement [89]–[91].
However, on the other hand it is sensitive to pattern variations
in positions, orientations and sizes. In [34], the columns of
each tactile matrix are concatenated to form a vector that is
directly treated as a descriptor, i.e., a “do-nothing” descriptor.
The feature is sensitive to the pose variances of objects, as a
result, a single object is assigned with multiple identities if
it is placed in different orientations to the robotic gripper. In
[35], the method using a “do-nothing” descriptor is taken as a
baseline and shows worse performance compared to the other
methods. In [92], the tactile readings of iCub fingertip with
12 taxels are taken to classify regions that the fingertip taps
into edges, plane and air.
• Statistical features: It is effortless to obtain statistical
features but the extracted features cannot be guaranteed to be
useful. In [93], statistical features are utilized, e.g., maximum,
minimum and mean pressure values of each tactile reading,
and positions of the center of gravity. As a result, the statistical
features of 52 different types form a 155 dimensional feature
vector. As the information that resides in some features is
redundant, an entropy based method is applied to investigate
the usefulness of the features but as a result low recognition
rate of only around 60% is achieved. The statistical features
have also been applied in other applications. In [94], [95],
touch attributes, e.g., pressure intensity and area of contact,
are taken as features to classify the touch modalities. In [96],
the internal states of bottles and cans are estimated using the
designed statical features of the tactile data.

y

where p and q stand for the order of the moment, x and y
stand for the horizontal and vertical positions of the cell in
the tactile image, respectively. In most cases, the moments
of order up to 2, i.e., (p + q) ∈ {0, 1, 2}, are computed and
other properties are based on them, like Hu’s moments used
in [35] and Zernike moments used in [88]. For example, in
[98], a tactile reading is approximated by an ellipse whose
principal axes are represented by the second order central
moments. Some other vision descriptors are also applied in
tactile sensing like regional descriptors used in [99]. The
Scale Invariant Feature Transform (SIFT) descriptor is created
based on image gradients [100] that has been proved robust to
cope with object pose variations in vision applications. This
is also particularly useful for tactile object recognition, since
the touches could introduce unexpected object rotation and
translation. SIFT has been explored in [35], [101]–[103] for
tactile recognition and a good performance can be achieved.
As both visual images and tactile readings are present in
numerical matrices, many other vision descriptors [104], [105]
also have the potential to be applied in tactile sensing, e.g.,
Shape Context [106], SURF [107] and 3D descriptor SHOT
[108].
Vision descriptors have also been applied in other tactile
applications [109]. Inspired by the similarities between tactile
patterns and grey-scale images, in [110] tactile data are transformed into histograms as structured features to discriminate
human-robot touch patterns. In [111], the pose of objects
placed on the sensor, i.e., the orientation of a cup handle
in this work, is estimated by applying Hough transform to
the impressed tactile profiles. In [112], the edge orientation is
estimated with a BioTac sensor by using a support vector regression (SVR) model. Image moments have also been widely
used in these applications. In [43], [113], image moments are
utilized to represent the acquired tactile data, applied in grasp
stability analysis. In a more recent work [114], in addition to
the image moments mentioned above, more haptic features

PREPRINT SUBMITTED TO ELSEVIER

including the 3D version of image moments and average
normal vector are taken into consideration for the same task,
together with features of other sensory streams, i.e., vision
and proprioception. In [115], both Hough transform and image
moments are employed and compared to predict the orientation
of an object edge so as to perform tactile servoing.
The GelSight tactile sensor has an extra-high resolution of
320×240 and provides the ability of using high level vision
descriptors [116]. It consists of a camera at the bottom and a
piece of clear elastomer on the top. The elastomer is coated
with a reflective membrane, which deforms to take the surface
geometry of the objects on it. The deformation is then recorded
by the camera under illumination from LEDs that project from
various directions. With the use of GelSight sensor, a multiscale Local Binary Pattern (LBP) descriptor is developed and
used to extract both micro- and macro-structures of the objects
from obtained tactile readings/images for texture classification
[68].
• PCA-based features: Principal Component Analysis (PCA)
can be applied to tactile readings and the acquired principal
components (PCs) are taken as features. It can reduce the
redundancy of tactile data and is easy to implement but it lacks
physical meaning [117]. In [37], a 16 × 16 tactile reading is
projected onto a feature space of a lower dimension and an
iterative procedure is taken to compute PCs with the largest
eigenvalues. In [38], the pressure distribution is defined as
M = [x y p]T , where x, y are location coordinates in
the sensor plane and p is the pressure in this location. By
covariance analysis of M , the resultant eigenvector lengths,
principal axis direction and shape convexity are taken as
features to recognize local shapes. The kernel PCA-based
feature fusion is used in [118] to fuse geometric features
and Fourier descriptors (based on Fourier coefficients) to
better discriminate objects. In [119], PCA is applied to reduce
the dimensionality of tactile readings. The obtained tactile
features are then used for grasp stability assessment and grasp
adaptation. In [120], PCA is also employed to extract features
from tactile readings that is applied for object pose estimation
[121].
• Self-organizing features: The aforementioned features are
predefined and hand-crafted and they are fed into shallow
classifiers, such as single-layer neural networks, kNN, and
SVM. Methods of this type can be easily implemented but
they can restrict the representation capability to serve different
applications and may only capture insignificant characteristics
for a task when using hand-crafted features [122]. In contrast,
there is no need to define the feature representation format
a-priori when using multilayer/deep architecture methods to
learn self-organizing features from raw sensor data. Soh et
al. [123] developed an on-line generative model that integrates a recursive kernel into a sparse Gaussian Process. The
algorithm iteratively learns from temporal tactile data and
produces a probability distribution over object classes, without constructing hand-crafted features. They also contribute
discriminative and generative tactile learners [124] based on
incremental and unsupervised learning. In [122] and [125],
unsupervised hierarchical feature learning using sparse coding
is applied to extract features from sequences of raw tactile

6

readings, for grasping and object recognition tasks. In [126]
denoising autoencoders with dropout are applied in tactile
object recognition and a dramatic performance improvement
of around 20% is observed in classifying 20 objects compared
to using shallow neural networks and supervised learning. In
[127], the randomized tiling Convolutional Network (RTCN)
is applied for the feature representation in tactile recognition
and can achieve an extremely good recognition rate of 100%
for most of the datasets tested in the paper. In [128], a joint
kernel sparse coding model is proposed for the classification
of tactile sequences acquired from multiple fingers. As new
techniques in deep learning and unsupervised learning emerge
and grow rapidly in recent years [129], it is promising to
apply more such algorithms to acquire self-organizing features
in tactile sensing. On the other hand, though deep learning
shows tremendous promise for the object recognition tasks,
online application of such a computationally intensive process
is difficult and it is hard to tune the parameters of deep
architectures; also, the complexity of these deep architectures
translate into difficulties in the introspection and physical
interpretation of the resulting model.
2) Discussions of local shape descriptors: A summary
of the discussed tactile features is given in Table III, with
discussions and comparison of pros and cons, and guidance on
the selection of tactile sensors. In the state-of-the-art literature,
vision based descriptors that take tactile readings as images are
widely employed and will also be the mainstream in extracting
features for tactile object recognition and other applications.
There is another trend to employ unsupervised learning and
deep architectures to learn self-organizing features from raw
tactile readings as an increasing number of such algorithms
are being developed [129]. In addition to shape recognition,
the various descriptors discussed here have also been used to
identify different contact patterns in several other tasks that
can be grouped as grasp stability assessment [43], [91], [113],
[114], [122], [130], identification of touch modalities [31],
[94], [110], object pose estimation [111], slip detection [109],
learning system states during in-hand manipulation [131],
contour following [92], tactile servoing [115], surface texture
recognition [68], localization and manipulation in assembly
tasks [116], and grasp adaptation [119]. They can also be
applied in other applications in the future research.
B. Global shape perception
The methods to recognize or reconstruct the global shape
of objects with tactile sensing can be grouped into three
categories with respect to the sensing inputs: 1) methods
using the distributions of contact points obtained from singlepoint contact force sensors or tactile sensors; 2) methods
based on analysing the pressure distributions in tactile arrays;
3) methods of combining both tactile patterns and contact
locations. Here the global shape refers to the overall shape of
objects, especially contours that extend beyond the fingertip
scale.
• Points based recognition: The methods of this type often
employ techniques from computer graphics to fit the obtained
cloud of contact points to a geometric model and outline

PREPRINT SUBMITTED TO ELSEVIER

7

TABLE III: A summary of previously studied tactile features
Feature type
Raw tactile
readings
Statistical
features

Advantages
A “do-nothing” descriptor; easy to implement; applied to any type of tactile data
Easy to be computed; based on statistics;
can be applied to any type of tactile data

Disadvantages
Lack of physical meaning; redundant; sensitive to variations of contact forces and poses
Lack of physical meaning; redundant; sensitive to force and pose variance; hand-crafted

Applicable sensors
Planar array sensors;
curved tactile sensors
Planar array sensors;
curved tactile sensors

Descriptors
adapted from
vision

Extract distinct features; invariant to force
and pose variance; can be used to share
information with vision

Some of the features are of high dimension
compared to original reading; hard to design; predefined and hand-crafted

Planar array sensors;
low curvature sensors

PCA-based
features
Selforganizing

Low dimensionality; statistics based; easy to
implement
No need to define the feature representation
format a-priori; learn features from raw data

Lack of physical meaning; sensitive to force
and pose variance
Lack of physical meaning; high computational complexity; hard to tune parameters

Planar
curved
Planar
curved

the object contour. This method was widely used by early
researchers due to the low resolution of tactile sensors and
prevalence of single-point contact force sensors [132]–[135].
In [133] resultant points from tactile readings are fit to superquadric surfaces to reconstruct unknown shapes. In a similar
manner, relying on the locations of contact points and hand
pose configurations, a polyhedral model is derived to recover
object shapes in [136]. These approaches are limited as objects
are usually required to be fixed and stationary. Different from
the point cloud based approaches, a non-linear model-based
inversion is proposed in [137] to recover surface curvatures by
using a cylindrical tactile sensor. In more recent works [138]–
[140], the curvatures at curve intersection points are analyzed
and thus a patch is described through polynomial fitting; in
[141], estimation of nonparametric shapes is demonstrated
using binary sensing (collision and no collision) and ergodic
exploration.
In some other works tactile sensors are utilized to classify
objects by taking advantage of the spatial distribution of the
object in space. In [142] an object representation is constructed
based on mosaics of tactile measurements, in which the
objects are a set of raised letter shapes. In this work, the
object recognition is regarded as a problem of estimating
a consistent location within a set of object maps and thus
histogram and particle filtering are used to estimate possible
states (locations). A descriptor based on the histogram of
triangles generated from three contact points was used in [143]
to classify 10 classes of objects. This descriptor is invariant
to object movements between touches but requires a large
number of samples (grasps) to accurately classify the touched
object. Kalman filters are applied in [144] to generate 3D
representations of objects from contact point clouds collected
by tactile sensors and the objects are then classified by the
Iterative Closest Point (ICP) algorithm. A similar method is
employed in [145], for haptic object recognition in underwater
environments. Through utilizing these methods, arbitrary contact shapes can be retrieved, however, it can be time consuming
when investigating a large object surface as excessive contacts
are required for recognizing the global object shape.
• Tactile patterns based recognition: Another approach is to
recognize the contact shapes using pressure distribution within
tactile arrays. As a result of the increasing performance of
tactile sensors, this approach has become increasingly popular
in recent years. Various methods to recognize the local contact

array sensors;
tactile sensors
array sensors;
tactile sensors

Ref.
[34], [91],
[92]
[64], [93]–
[95]
[35], [81],
[88],
[97],
[111], [113],
[114]
[38], [119],
[121]
[122]–[124],
[126], [127]

shapes have been reviewed and discussed in Section IV-A.
In terms of recognising the global object shape by analysing
pressure distributions in tactile images collected at different
contact locations, however, a limited number of approaches
are available. One popular method is to generate a codebook
of tactile features and use it to classify objects and a particular
paradigm is the Bag-of-Features (BoF) model [34], [35], [122].
The BoF model originates from the Bag-of-Words (BoW)
model in natural language processing for text classification and
has been widely utilized in the field of computer vision [146],
thanks to the simplicity and power of the model. Inspired by
the similar essence of vision and tactile sensing, Schneider
et. al. [34] first applied the BoF model in tactile object
recognition. In this framework, local contact features extracted
from tactile readings in the training phase are clustered to form
a dictionary and cluster centroids are taken as “codewords”.
Based on the dictionary, each tactile feature is assigned to its
nearest codeword and a fixed length feature occurrence vector
is generated to represent the object. It is easy to implement and
can achieve an appropriate performance [34], [35], however,
only local contact patterns are taken and the distribution of
the features in three-dimensional space is not incorporated.
• Object recognition based on both sensing modalities:
For humans, the sense of touch consists of both kinaesthetic
and cutaneous sensing and these two sensing modalities are
correlated [2]. Therefore, the fusion of the spatial information
and tactile features could be beneficial for the object recognition tasks. This combination has already been proved to
improve recognition capabilities in teleoperation experiments
with human subjects both in identifying curvatures [147]
and estimating stiffness [148]. In [149], [150], a series of
local “tactile probe images” is assembled and concatenated
together to obtain a “global tactile image” using 2D correlation
techniques with the assistance of kinaesthetic data. However,
in this work tactile features are not extracted whereas raw
tactile readings are utilized instead, which would bring high
computational cost when investigating large object surfaces.
In [151], three different models are proposed based on proprioceptive and tactile data, using Self-Organising Maps (SOMs)
and Neural Networks. In [117], the tactile and kinaesthetic
data are integrated by decision fusion and description fusion
methods. In the former, classification is done with two sensing
modalities independently and recognition results are combined
into one decision afterwards. In the latter, the descriptors of

PREPRINT SUBMITTED TO ELSEVIER

kinaesthetic data (finger configurations/positions) and tactile
features for a single palpation are concatenated into one
vector for classification. In other words, the information of
the positions where specific tactile features are collected is
lost. In both methods, the tactile and kinaesthetic information
is not fundamentally linked. In a similar manner, in [152]
the tactile and kinaesthetic modalities are fused in a decision
fusion fashion. Both tactile features and joint configurations
are clustered by SOMs and classified by ANNs separately
and the classification results are merged to achieve a final
decision. In a more recent work [153], the actuator positions of
robot fingers and force values of embedded TakkTile sensors
form the feature space to classify object classes using random
forests but there are no exploratory motions involved, with
data acquired during a single and unplanned grasp. In [154],
an algorithm named Iterative Closest Labeled Point (iCLAP)
is proposed to recognize objects using both tactile and kinaesthetic information that has been shown to outperform those
using either of the separate sensing modalities. In general, it
is still an open question how to link both sensor locations and
tactile images.
In summary, different strategies have been taken to recover
the global shape of objects in the view of proprioception and
tactile sensing, as summarized in Table IV with a comparison
of pros and cons of different methods. Some researchers
take advantage of the distributions of contact points in space,
e.g., points based methods, whereas some others utilize local
patterns only, e.g., BoF framework; and it is also expected to
achieve a better perception by combining both distributions of
contact points in space and local contact patterns.
V. P OSE ESTIMATION VIA TOUCH SENSING
Effective object manipulation requires accurate and timely
estimation of the object pose. This pose is represented by the
object’s position and orientation with respect to the robot endeffector or to a global coordinate frame. Even small errors
the estimate of the object’s location can lead to incorrect
placement of the robot fingers on the object, generate wrong
assumptions on grasp stability and compromise the success
of a manipulation task. In fact, in-hand manipulation is, by
definition, the task of changing an object’s pose from an initial
to a final configuration [155]. Thus, robust, accurate and fast
perception of an object’s pose must be a crucial part of any
sophisticated grasping and manipulation system.
The most common means in robotics to estimate an object’s
pose is using computer vision. However, when a robot approaches the object to be manipulated, it creates occlusions
and vision cannot be relied upon. To cope with this problem,
tactile sensing has been used to assist a robot in determining the pose of a touched object, either on its own or in
combination with a vision system. In this review we classify
existing techniques according to the sensing inputs: singlepoint contact sensor and tactile sensing arrays, on their own or
used together with a vision system (contact-visual and tactilevisual), as shown in Table V.
• Single-point contact based: Due to the poor performance
of tactile sensors, most early works tend to use single-point

8

contact sensors, i.e., force sensors and dynamic tactile sensors,
for localizing the objects or features. Early work on finding
an object’s pose used only angle and joint-torque sensing and
used an interpretation tree that contained possible correspondences between object vertices to fingers [156]. In [157], a
force-controlled robot is used to localize objects using Markov
Localization that is applied in a task of inserting a cube
(manipulated object) into a corner (environment object) by a
manipulator. This compliant motion problem is also compared
with the data association [158], i.e., assigning measurements
to landmarks, and global localization with different models
in Simultaneous Localization And Mapping (SLAM) for mobile robotics. In both cases, Bayesian based approaches can
provide a systematic solution to estimate both models and
states/parameters simultaneously. In [159], a set of algorithms
are implemented for SLAM during haptic exploration but only
simulation results are presented.
Particle filtering is popular in (optical/acoustic-based) robot
localization problems and may merit further investigation in
tactile sensing, where objects could be modelled as clouds
of particles distributed according to iteratively updated probability distributions. In [160], particle filtering is applied to
estimate the object pose and track the hand-object configurations, especially in the cases where objects are possibly
moving in the robot hand. Also using particle filtering, small
objects, e.g., buttons and snaps, are localized and manipulated
in flexible materials such as fabrics that are prone to move
during robot manipulations in [161]. Improvements of the
particle filter algorithm have been presented to address the
problem of localizing an object via touch. In [162]–[164], this
novel particle filter, named Scaling Series, has each particle
representing not a single hypothesis, but a region in the search
space which is sequentially refined. Besides, annealing is
used to improve sampling from the measurement model. This
method was tested using a robot manipulator equipped with
a 6D force/torque sensor. It is applied in two scenarios: 1) to
localize (estimate positions and orientations), grasp and pick
up a rectangular box; 2) to grasp a door handle. A method
that includes memory from past measurements and a “scaled
unscented transformation” is performed on the prediction step
was presented in [165].
A “haptic map” is created from the proprioceptive and contact
measurements during the training that is then used to localize
the small objects embedded in flexible materials during robot
manipulation. By using dynamic tactile sensors inspired by rat
whiskers, the grid based SLAM is introduced in [16] to navigate a robot with touch sensors by deriving timing information
from contacts and a given map about edges in a small arena, in
which particle filters are also used. In [166], SLAM is applied
to recover the shape and pose of a movable object from a
series of observed contact locations and contact normals with
a pusher. Analogous to SLAM in mobile robotics, the object
is taken as a rigid but moving environment and the pusher is
taken as a sensor to get noisy observations of the locations
of landmarks. However, compared to the exploration using
visual feedback, tactile exploration is challenging in the sense
that touch sensing is intrusive in nature, that is, the object
(environment) is moved by the action of sensing.

PREPRINT SUBMITTED TO ELSEVIER

9

TABLE IV: Methods for global shape recognition
Modalities
Contact
points
Tactile
patterns
Both
sensing
modalities

Recognition methods
Graphical models (point clouds)
[136]; polynomial fitting (surface
curvatures) [140]; filtering (spatial
distribution) [142], [144]
Bag-of-Features
[34], [35], [102], [122]
Image stitching [149], [150]; Decision fusion [117], [152]; Description fusion [117]

Advantages
Arbitrary shapes can be retrieved; object graphical models can be built; spatial distribution is revealed
Easy to implement; local shape
features are employed
Both kinaesthetic and cutaneous cues are included

TABLE V: Different sensing inputs for object pose estimation
Types
Visual
Contact
Tactile
Contact-Visual
Tactile-Visual

Sensors involved
Cameras, lasers
Single-point
contact
sensors
Tactile sensors
Cameras, single-point
contact sensors
Cameras, tactile sensors

Information
Global

Example
[167]

Local

[164]

Local

[116]

Global+local

[168]

Global+local

[169]

Note: Laser scanners and cameras (widely used in mobile robots) collect
global information about the environment, whereas the contact sensors
(single-point contact sensors and tactile sensors) provide local information.

• Tactile-based: The increasing performance of tactile sensors
provides the feasibility to localize objects or object features in
robot hand using the information derived from tactile arrays.
With the high-resolution GelSight sensor, collected tactile
images can be localized [116] within a height map via image
registration to help localize objects in hand. A height map is
first built based on the collected tactile readings. The keypoints
are then localized from both the map and incoming tactile
measurements. After that, feature descriptors are extracted
from both and matched. In this manner, the pose of an
assembly part in the robot hand can be estimated. Similarly to
the way that local geometric features can be extracted using
PCA, these features were used to determine an object’s pose.
In [170] a Monte Carlo method was used to find an object
pose where the local geometry of the object at the contact
location matched the PCA features obtained from the tactile
sensor. Another approach relied on the fact that, when the
robot is in contact with the object, the possible locations of
the object must lie inside a contact manifold, a novel particle
filter was developed that was both faster and more accurate
than a standard particle filter [171].
• Contact-Visual based: Vision and tactile sensing share
information of how objects are present in 3D space in the
form of data points. For vision, a mesh can be generated
based on point cloud from 3D cameras or laser scanners
that can also be obtained by contact sensors (in this case,
mostly single-point contact sensors). In [172], localizing object
within hand is treated as a hybrid estimation problem, by
fusing data from stereo vision, force-torque and joint sensors.
Though reasonable object localization can be achieved, tactile
information on the fingertips is not utilized, being however
promising to obtain better performance of object localization
by incorporating additional information. Based on the assumption that visually similar surfaces are likely to have similar

Disadvantages
Time consuming when investigating large surfaces; excessive contacts required; local
features are not revealed
The distribution of the features
in 3D space is not incorporated
Hard to associate local patterns
and kinaesthetic data; bring
additional computational cost

Applicable sensors
Single-point
contact
sensors;
planar/curved
tactile sensors
Planar sensors or
low curvature
Planar sensors or
low curvature

haptic properties, vision is used to create dense haptic maps
efficiently across visible surfaces with sparse haptic labels in
[168]. Vision can also provide an approximate initial estimate
of the object pose that is then refined by tactile sensing using
local [173], [174] or global optimization [170].
• Tactile-Visual based: Vision and tactile sensing of humans
have also been found to share the representations of objects
[175]. To put it in another way, there are some correspondences, e.g., prominent features, between vision and tactile
sensing while observing objects. In [169], it is proposed to
localize a tactile sensor in a visual object map by sharing similar sets of features between visual map and tactile readings.
It is treated as a probabilistic estimation problem and solved
in a framework of recursive Bayesian filtering, where tactile
patterns are for local information and vision is to provide a
global map to localize sensor contact on the object.
In summary, the localization of objects during manipulation
using tactile sensing remains an open problem. While the
most popular approaches rely on techniques similar to the
ones used in SLAM applications, high accuracy and realtime performance is yet to be achieved. Furthermore, these
approaches require a large number of contacts, which may be
impractical for effective object manipulation. Nevertheless, it
is foreseen that tactile sensors of different types will need to
be used along with vision for solving the localization problem.
Due to the reduced number of contacts, high resolution tactile
sensors that can capture the object’s local shape can be useful
to determine the pose using few contacts.
VI. TACTILE SENSING IN SENSOR FUSION
Robots must be equipped with different sensing modalities to
be able to operate in unstructured environments. The fusion of
these different sources of data into more meaningful, higherlevel representations of the state of the world is also part of
the process of perception (See Fig. 1). The information that
is provided by the sensors may be redundant, reducing the
uncertainty with which features are perceived by the system,
and increasing the reliability of the system in case of failure
[176]. Combining different sensing modalities which provide
complementary information may also have a synergistic effect,
where features in the environment can be perceived in situations where it would be impossible using the information from
each sensor separately [177]. Furthermore, multiple sensors
can provide more timely and less costly information, given
the different operating speeds of the sensors and the fact that
their information can be processed in parallel.
In tasks that require interaction with the environment, tactile

PREPRINT SUBMITTED TO ELSEVIER

sensing can be combined with other sensing modalities to
increase precision and robustness [178]. Most typical sensing
arrangements in robot manipulation and other tasks that include physical interaction, are combinations of tactile sensing
with vision, kinaesthetic cues, force-torque and range sensing
[179]. We have discussed the works on integrating tactile
sensing and kinaesthetic cues for global shape recognition
in Section IV-B. In this section, we mainly focus on tactile
sensing in sensor fusion with vision.
In robotics, attempts to fuse vision and touch to recognize
and represent objects can be dated back to the 1980’s [180]. In
most applications, tactile sensing was utilized to support vision
for improving performance in object recognition [181], object
reconstruction [182] and grasping [114]. With the increasing
performance in the last few decades, tactile sensors have
shown the potential to play a more significant role in tasks
using information integrated from different modalities [172],
[183], [184]. Thanks to the development of tactile sensor
technologies, the role of tactile sensing in sensor fusion for
multiple applications has evolved to a mature stage in a
number of applications, which can be summarized as follows:
• Verifying contacts: Most early researchers took tactile
sensors as devices to verify contacts due to their low resolution. In this type of methods, rough object models are
first built by vision and the description is then refined and
detailed by tactile sensing. For instance, in [180] vision is
first used to obtain object contours and edges as it can capture
rich information rapidly, by taking bi-cubic surface patches
as primitives. The tactile trace information is then added into
the boundary curves acquired in the first step to achieve a
more detailed description of the surfaces. In this work, tactile
(haptic) sensing is utilized to trace the object surface to get
3D coordinates, surface normal and tangential information
of contact points. A similar framework is also employed in
[185], where the role of tactile sensing is greater. The strategy
consisted of exploring regions that are uncertain for vision,
determining the features such as surfaces, holes or cavities, and
outlining the boundary curves of these features. In [186] vision
is used to estimate the contact positions along a finger and
applied forces for grasping tasks, whereas tactile sensors and
internal strain gauges are utilized to assist vision, especially
in two cases: when vision is occluded or when it is needed to
further confirm contact positions determined by vision.
In [187], implicit surface potentials described by Gaussian
Processes are taken as object models for internal shape representations. In the Gaussian Process of shape estimation,
uncertain sensor channels, i.e., tactile, visual and laser, are
equally integrated, to support the control of reach-and-grasp
movements. In [182], an optimal estimation method is proposed to learn object models during grasping via visual and
tactile data by using iterative extended Kalman filter (EKF).
A similar work is carried out in [181] but Gaussian process
regression is utilized instead of EKF: visual features are
extracted first to form an initial hypothesis of object shapes;
tactile measurements are then added to refine the object model.
Though tactile array sensors are employed in these works,
still only the locations of tactile elements that are in contact
with the object are used, not including the information of

10

pressure distribution in tactile arrays. A framework to detect
and localize contacts with the environment using different
sources of sensing information is presented in [176]. Each
sensor generates contact hypotheses which are fused together
using a probabilistic approach to obtain a map that contains
the likelihood of contact at each location in the environment.
This approach was tested in different platforms with various
sensing modalities, such as tactile, force-torque, vision, and
range sensing.
• Extracting features to assist vision: In [114], multiple
sensory streams, i.e., vision, proprioception and tactile sensing,
are integrated to facilitate grasping in a goal-oriented manner.
In this work, image moments of tactile data (both 2D and 3D),
are utilized; together with vision and action features, a feature
set is formed for grasping tasks. In [172], the in-hand object
location is estimated by fusing data from stereo vision, forcetorque and joint sensors. The sensor fusion is achieved by
simply concatenating features or data from single modalities
into a joint vector. In [188], vision and tactile sensing are
proved to be complementary in the task of identifying the
content in a container by grasping: as the container is squeezed
by a robot hand, deformation of the container is observed by
vision and the pressure distributions around the contact region
are captured by tactile sensors. PCA is employed to extract
PCs from vision, tactile and vision-tactile data, which are
then fed into classifiers. It is concluded that combining vision
and tactile sensing leads to the improvement of classification
accuracy where one of them is weaker. In addition, some
researchers have attempted to combine vision, tactile and force
sensing to estimate the object pose in [170].
• Providing local and detailed information: In [98], a multisensor control framework is created for a task of opening a
door. The vision is used to detect the door handle prior to
the gripper-handle contact and tactile sensors mounted on the
robot grippers are used to measure the gripper orientation with
respect to the door handle by applying moment analysis to
tactile readings during the gripper-handle contact. Based on the
information, the pose of grippers can be adjusted accordingly.
In [189], vision, force/torque sensors and tactile sensors are
also combined in a control scheme for opening a door. A
sensor hierarchy is established in which the tactile sensing
is preferred over vision: the tactile sensing can obtain robust
and detailed information about the object position, whereas
vision provides global but less accurate data. In both works,
tactile sensors play a greater role than just confirming contacts
but the vision and tactile sensing are employed in different
phases instead of achieving a synthesized perception. In recent
work [190], a GelSight sensor is taken as a source of dense
local geometric information that is incorporated in object point
cloud from an RGB-D camera.
• Transferring knowledge with vision: The information of
object representations has been found to be shared by vision
and tactile sensing of humans [175] and the visual imagery
is discovered to be involved in the tactile discrimination of
orientation in normally sighted humans [191]. It has also been
shown that the human brain employs shared models of objects
across multiple sensory modalities such as vision and tactile
sensing so that knowledge can be transferred from one to

PREPRINT SUBMITTED TO ELSEVIER

another [192]. This mechanism can also be applied in robotic
applications. In [15], vision and tactile samples are paired
to classify materials using a dynamic tactile sensor. In the
training phase, material representations are learned by both
tactile (haptic) and visual observations of object surfaces. A
mapping matrix for transferring knowledge from vision to
tactile domain is then learned by dimensionality reduction and
at the test stage, materials can be classified with only tactile
(haptic) information available based on the obtained shared
models. In [193], vision and tactile feedback are associated by
mapping visual and tactile features to object curvature classes.
By taking advantage of the fact that both visual and tactile
images are present in numerical matrices, a tactile sensor can
be localized in a visual object map by sharing the same sets
of features between the visual map and tactile readings [169].
In recent works [194] and [195], deep networks, typically
Convolutional Neural Networks (CNNs), are applied to both
vision and hatpic data and the learned features from both
modalities are then associated by a fusion layer for haptic
classification.
To conclude, as the performance of tactile sensors improves,
tactile sensing plays an increasingly important role in the
sensor fusion for multiple applications. In addition to providing information of contact locations like single-point contact sensors, more useful cues can be extracted from tactile
readings, such as detailed local information and features to
improve models built by vision. More importantly, knowledge
can be transferred between vision and tactile sensing, which
can improve the perception of the environment interacting with
robots.
VII. D ISCUSSION AND CONCLUSION
The rapid development of tactile devices and skins over
the last couple of years has provided new opportunities for
applying tactile sensing in various robotic applications. Not
only the types of tactile sensors are to be considered for an
application, the suitable computational method to decipher the
encoded information is also of the utmost importance. At the
same time it has brought to the fore numerous challenges
towards effective use of tactile data. This paper focuses on
tactile object recognition of shape and surface material and
the estimation of its pose. Knowledge of these properties is
crucial for the successful application of robots in grasping and
manipulation tasks. In this section the progress in the literature
is summarised, open issues are discussed and future directions
are highlighted.
• Selection of tactile sensors to meet the requirements
of tactile object recognition and localization: Apart from
the algorithms used to interpret sensor data, the recognition
performance also depends on what tactile sensors are used. As
high spatial and magnitude resolution provides more detailed
information on the object, tactile sensors of higher resolution
are preferable. But on the other hand, higher resolution will
likely bring about higher costs, both in development and
fabrication of the sensor. Most of the available tactile sensors
used in robotic fingertips are of lower or similar resolution
over an equivalent area compared to the spatial resolution of

11

the human finger, i.e., the density of Merkel receptors in the
fingertip (14×14) [35]. For instance, the widely used tactile
sensor modules in the literature Weiss DSA 9205 (14×6)
with a size of 24 mm by 51 mm; the Weiss DSA 9335
sensor module provides a similar spatial resolution of 3.8 mm
with 28 sensor cells organized as 4×7 matrices; the TakkTile
array sensor senses independent forces over an 8×5 grid with
approximately 6 mm resolution [21]; the DigiTacts sensor
from Pressure Profile Systems has 6×6 sensor elements with
6×6 mm resolution; a Tekscan sensor (model: 4256E) has a
sensing pad of 24 mm by 35 mm with 9×5 elements. To
achieve good spatial resolution, these sensors are commonly
costly. An exception is the GelSight tactile sensor that has high
resolution of 320×240 and is economical, as it utilizes a lowcost webcam to capture the object properties. But the inclusion
of a webcam and illumination LEDs makes the whole sensor
bulky and difficult to miniaturize. Therefore, researchers need
to balance multiple factors when selecting tactile sensors, i.e.,
spatial and magnitude resolution, size and cost of the sensors.
In real-time tactile localization applications, it is required that
tactile sensors have quick responses and short latency. Human
tactile receptors can sense the transient properties of a stimulus
at up to 400 Hz [84]. Since the output of single-point contact
sensors, i.e., force sensors and whisker based sensors, is of
low dimension, these sensors have typically high update rate
and low latency. For instance, the data-acquisition of strain
gauges and PVDFs embedded in a robotic finger can achieve
a temporal resolution of 2.5 kHz [84]. In contrast, tactile array
sensors output data of higher resolution and therefore need to
reduce the latency and improve response times. The maximum
scanning rate for the Weiss DSA 9205 sensor is around 240
frames per second (fps) [34], while using a serial bus (RS-232)
shared by multiple sensors for data transmission will limit the
available frame rate for each sensor to approximate 30 Hz
when six sensor pads are used [144]; the update rate of the
Tekscan pressure sensing is set as 200 Hz in [90]; by using
a GelSight sensor, the in-hand localization can be achieved
at 10 fps [116]. An alternative method to sense the transient
stimulus is to use neuromorphic tactile systems that provides
transient event information as the sensor interface with objects
[196], [197]. The sensing element triggers an event when it
detects relevant transient stimulus and this provides important
local information in manipulation [198].
Many of the tactile sensors used in the literature are of
rigid sensing pad and plated with an elastic cover such as
Weiss modules and TakkTile sensors. As the data are read
from sensing elements arranged in a plane, it reduces the
complexity of data interpretation but these sensors are hard
to be mounted on curved robot body parts. There are also
flexible tactile sensors like DigiTacts from PPS and Tekscan
tactile system. They can be attached to curved surfaces but the
data interpretation becomes more complex.
• Tactile object recognition: As for the local shape recognition by touch sensing, multiple feature descriptors have been
proposed in the literature to represent features within tactile
readings. The “do-nothing” descriptors, statistical features,
and PCA-based features are simple and easy to implement
but normally sensitive to pattern variances. Vision-based de-

PREPRINT SUBMITTED TO ELSEVIER

scriptors have been observed to be the mainstream by taking
tactile readings as images. It can be foreseen that more vision
descriptors will be applied in touch sensing to represent the
information embedded in the tactile data which may provide
cues of how to combine vision and touch sensing. In addition,
unsupervised learning and deep architectures show promise in
recent years by obtaining self-organizing features from raw
tactile readings without pre-defining the feature structure. To
seek shape representation appropriate for tactile sensing, these
methods originated from vision could be further investigated
by examining the differences between vision and tactile sensing mechanisms.
To reveal the global object shapes, one method is to utilize
the distribution of contact points that can retrieve arbitrary
contact shapes but will be time consuming when investigating
a large object surface, while also being sensitive to object
movements. The other method is to use local tactile patterns
and a paradigm of this method is Bag-of-Features that is
becoming popular recently. It can reveal local features but
the three-dimensional distribution information of the object
is not incorporated. One future direction is to perceive global
shapes by integrating tactile patterns and kinaesthetic cues to
achieve a better recognition performance as cutaneous sensing
and proprioception are correlated in object recognition.
• Object localization by touch sensing: The object localization via touch could be a significant application that can
facilitate object manipulation and can be complementary to visual localization. Currently, most of the attention regarding this
topic has been paid to haptic-based localization with singlepoint contact sensors. But single-point contact sensors, like
force sensors and whiskers, provide only limited information,
i.e., contact locations of single points. Single-point contact
sensors also require multiple contacts, which may cause the
unpredicted motion of the object during the contacts. Thanks
to the advancement of tactile devices, the works into other
types of localization have emerged, especially using tactile
array sensors to identify contact patterns. There exist multiple
open issues expected to be investigated: 1) how to map the
contact points on the sensor pad to the 3D space; 2) how
to combine the data points acquired from vision and tactile
sensing; 3) how to define the priority hierarchy when the
information from different sensors conflicts.
• Tactile sensing in sensor fusion: The use of tactile
sensing has also been applied to material recognition and
some other emerging applications such as slip detection, grasp
stability assessment, identification of touch modalities and
tactile servoing. The role of tactile sensing plays in sensor
fusion has evolved from just verifying object-sensor contacts
to extracting features to assist vision and provide detailed
local information, and to transferring knowledge with vision.
It is envisioned that tactile sensing will play a more important
role in multiple applications. The development of a “sensor
hierarchy” could be sought to establish the “preference” of
sensor data.
• Deep learning in robotic tactile perception: After dominating the computer vision field, deep learning, especially
CNNs, has also been applied in robotic tactile perception
in recent years, from object shape recognition [126], [127],

12

hardness estimation [79], to sharing features with vision [194],
[195]. When using deep learning, there are several issues
need to be carefully taken care of. 1) Why deep learning?
Deep learning can learn self-organising features instead of
defining hand-crafted features a-priori and has shown considerable success in tasks such as object recognition. However,
online application of such computationally intensive algorithm
is difficult, especially for tasks need timely response such
as grasping and manipulation, and it is hard to tune the
parameters of deep architectures. On the other hand, it would
be an overkill to put deep learning on small datasets and
simple tasks; such algorithms can also potentially lose physical
interpretation. 2) Data collection. Compared to datasets in
vision, it is relatively expensive to collect tactile data, not
only because of the required robot components but also the
efforts that go into designing the data collection process.
On the other hand, the same algorithm may achieve distinct
performances on data collected from different tactile sensors.
Due to such reasons, there are still no commonly used datasets
for evaluating the processing algorithms in tactile sensing but
a substantially large haptic dataset would benefit the field. 3)
Labeling and ground truth. It is hard to obtain ground truth
for tactile perception tasks using crowd-sourcing tools like
Amazon Mechanical Turk in vision. In the current literature
[194], [195], tactile properties such as haptic adjectives (e.g.,
compressible or smooth) are labeled by limited number of
human annotators which may bring bias to the ground truths.
To this end, domain adaptation can be applied from vision to
tactile sensing and unsupervised learning will be of benefit in
the future research of tactile sensing.
ACKNOWLEDGMENT
The work presented in this paper was partially supported
by the Engineering and Physical Sciences Council (EPSRC)
Grant (Ref: EP/N020421/1), the King’s-China Scholarship
Council PhD scholarship, the European Commission under
grant agreements PITNGA2012317488-CONTEST, and EPSRC Engineering Fellowship for Growth Printable Tactile
Skin (EP/M002527/1).
R EFERENCES
[1] R. S. Dahiya and M. Valle, Robotic tactile sensing: technologies and
system. Springer Science & Business Media, 2012.
[2] S. J. Lederman and R. L. Klatzky, “Haptic perception: A tutorial,”
Atten. Percept. Psychophys., vol. 71, no. 7, pp. 1439–1459, 2009.
[3] R. S. Dahiya, P. Mittendorfer, M. Valle, G. Cheng, and V. J. Lumelsky,
“Directions toward effective utilization of tactile skin: A review,” IEEE
Sensors J., vol. 13, no. 11, pp. 4121–4138, 2013.
[4] R. S. Dahiya, G. Metta, M. Valle, and G. Sandini, “Tactile sensing–
from humans to humanoids,” IEEE Trans. Robot. (T-RO), vol. 26, no. 1,
pp. 1–20, 2010.
[5] A. Schmitz, P. Maiolino, M. Maggiali, L. Natale, G. Cannata, and
G. Metta, “Methods and technologies for the implementation of largescale robot tactile sensors,” IEEE Trans. Robot. (T-RO), vol. 27, no. 3,
pp. 389–400, 2011.
[6] M. Kaltenbrunner, T. Sekitani, J. Reeder, T. Yokota, K. Kuribara,
T. Tokuhara, M. Drack, R. Schwödiauer, I. Graz, S. Bauer-Gogonea
et al., “An ultra-lightweight design for imperceptible plastic electronics,” Nature, vol. 499, no. 7459, pp. 458–463, 2013.
[7] N. Yogeswaran, W. Dang, W. T. Navaraj, D. Shakthivel, S. Khan, E. O.
Polat, S. Gupta, H. Heidari, M. Kaboli, L. Lorenzelli et al., “New
materials and advances in making electronic skin for interactive robots,”
Adv. Robotics, vol. 29, no. 21, pp. 1359–1373, 2015.

PREPRINT SUBMITTED TO ELSEVIER

[8] S. Khan, L. Lorenzelli, and R. S. Dahiya, “Technologies for printing
sensors and electronics over large flexible substrates: a review,” IEEE
Sensors J., vol. 15, no. 6, pp. 3164–3185, 2015.
[9] B. Siciliano and O. Khatib, Springer handbook of robotics. Springer,
2016.
[10] B. D. Argall and A. G. Billard, “A survey of tactile human-robot
interactions,” Robot. Auto. Syst., vol. 58, no. 10, pp. 1159–1176, 2010.
[11] Z. Kappassov, J. A. Corrales, and V. Perdereau, “Tactile sensing in
dexterous robot hands – review,” Robot. Auto. Syst., vol. 74, pp. 195–
220, 2015.
[12] R. S. Dahiya, D. Cattin, A. Adami, C. Collini, L. Barboni, M. Valle,
L. Lorenzelli, R. Oboe, G. Metta, and F. Brunetti, “Towards tactile
sensing system on chip for robotic applications,” IEEE Sensors J.,
vol. 11, no. 12, pp. 3216–3226, 2011.
[13] R. Koiva, M. Zenker, C. Schürmann, R. Haschke, and H. J. Ritter, “A
highly sensitive 3d-shaped tactile sensor,” in Proc. IEEE/ASME Int.
Conf. Advanced Intelligent Mechatronics (AIM), 2013, pp. 1084–1089.
[14] G. H. Büscher, R. Kõiva, C. Schürmann, R. Haschke, and H. J. Ritter,
“Flexible and stretchable fabric-based tactile sensor,” Robot. Auto. Syst.,
vol. 63, pp. 244–252, 2015.
[15] O. Kroemer, C. H. Lampert, and J. Peters, “Learning dynamic tactile
sensing with robust vision-based training,” IEEE Trans. Robot. (T-RO),
vol. 27, no. 3, pp. 545–557, 2011.
[16] C. Fox, M. Evans, M. Pearson, and T. Prescott, “Tactile SLAM with a
biomimetic whiskered robot,” in Proc. IEEE Int. Conf. Robot. Autom.
(ICRA), 2012, pp. 4925–4930.
[17] K. J. Kuchenbecker, J. Fiene, and G. Niemeyer, “Improving contact realism through event-based haptic feedback,” IEEE Trans. Vis. Comput.
Graphics (VCG), vol. 12, no. 2, pp. 219–230, 2006.
[18] B. Mitchinson, M. J. Pearson, A. G. Pipe, and T. J. Prescott,
“Biomimetic tactile target acquisition, tracking and capture,” Robot.
Auto. Syst., vol. 62, no. 3, pp. 366–375, 2014.
[19] L. A. Huet, J. W. Rudnicki, and M. J. Hartmann, “Tactile sensing with
whiskers of various shapes: Determining the three-dimensional location
of object contact based on mechanical signals at the whisker base,” Soft
Robotics, vol. 4, no. 2, pp. 88–102, 2017.
[20] H. Xie, H. Liu, S. Luo, L. D. Seneviratne, and K. Althoefer, “Fiber
optics tactile array probe for tissue palpation during minimally invasive
surgery,” in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS), 2013,
pp. 2539–2544.
[21] Y. Tenzer, L. P. Jentoft, and R. D. Howe, “The feel of MEMS
barometers: inexpensive and easily customized tactile array sensors,”
IEEE Robot. Autom. Mag., vol. 21, no. 3, pp. 89–95, 2014.
[22] C. Chorley, C. Melhuish, T. Pipe, and J. Rossiter, “Development of a
tactile sensor based on biologically inspired edge encoding,” in Proc.
IEEE Int. Conf. Adv. Robot. (ICAR), 2009, pp. 1–6.
[23] K. Sato, K. Kamiyama, N. Kawakami, and S. Tachi, “Finger-shaped
gelforce: sensor for measuring surface traction fields for robotic hand,”
IEEE Trans. Haptics, vol. 3, no. 1, pp. 37–47, 2010.
[24] M. K. Johnson, F. Cole, A. Raj, and E. H. Adelson, “Microgeometry
capture using an elastomeric sensor,” ACM Trans. Graph. (TOG),
vol. 30, no. 4, pp. 46–53, 2011.
[25] A. Yamaguchi and C. G. Atkeson, “Combining finger vision and optical
tactile sensing: Reducing and handling errors while cutting vegetables,”
in Proc. IEEE-RAS Int. Conf. Human. Robot. (Humanoids), 2016, pp.
1045–1051.
[26] E. O. Polat, O. Balci, N. Kakenov, H. B. Uzlu, C. Kocabas, and
R. Dahiya, “Synthesis of large area graphene for high performance
in flexible optoelectronic devices,” Scientific Reports, vol. 5, p. 16744,
2015.
[27] M. Hoffmann, Z. Straka, I. Farkas, M. Vavrecka, and G. Metta,
“Robotic homunculus: Learning of artificial skin representation in a
humanoid robot motivated by primary somatosensory cortex,” IEEE
Trans. Cog. Devel. Sys., 2017.
[28] L. Muscari, L. Seminara, F. Mastrogiovanni, M. Valle, M. Capurro,
and G. Cannata, “Real-time reconstruction of contact shapes for large
area robot skin,” in Proc. IEEE Int. Conf. Robot. Autom. (ICRA), 2013,
pp. 2360–2366.
[29] P. Mittendorfer and G. Cheng, “Humanoid multimodal tactile-sensing
modules,” IEEE Trans. Robot. (T-RO), vol. 27, no. 3, pp. 401–410,
2011.
[30] C. Bartolozzi, L. Natale, F. Nori, and G. Metta, “Robots with a sense
of touch,” Nature Materials, vol. 15, no. 9, pp. 921–925, 2016.
[31] M. Kaboli, A. Long, and G. Cheng, “Humanoids learn touch modalities
identification via multi-modal robotic skin and robust tactile descriptors,” Adv. Robotics, vol. 29, no. 21, pp. 1411–1425, 2015.

13

[32] W. Dang, V. Vinciguerra, L. Lorenzelli, and R. Dahiya, “Printable
stretchable interconnects,” Flexible and Printed Electronics, vol. 2,
no. 1, p. 013003, 2017.
[33] H. Heidari, N. Wacker, and R. Dahiya, “Bending induced electrical
response variations in ultra-thin flexible chips and device modeling,”
Applied Physics Reviews, vol. 4, no. 3, p. 031101, 2017.
[34] A. Schneider, J. Sturm, C. Stachniss, M. Reisert, H. Burkhardt, and
W. Burgard, “Object identification with tactile sensors using Bag-ofFeatures,” in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS),
2009, pp. 243–248.
[35] Z. Pezzementi, E. Plaku, C. Reyda, and G. D. Hager, “Tactile-object
recognition from appearance information,” IEEE Trans. Robot. (T-RO),
vol. 27, no. 3, pp. 473–487, 2011.
[36] G. Cannata, S. Denei, and F. Mastrogiovanni, “A framework for
representing interaction tasks based on tactile data,” in Proc. IEEE
Int. Work. Robot Human Inter. Commun., 2010, pp. 698–703.
[37] G. Heidemann and M. Schopfer, “Dynamic tactile sensing for object
identification,” in Proc. IEEE Int. Conf. Robot. Autom. (ICRA), 2004,
pp. 813–818.
[38] H. Liu, X. Song, T. Nanayakkara, L. D. Seneviratne, and K. Althoefer,
“A computationally fast algorithm for local contact shape and pose
classification using a tactile array sensor,” in Proc. IEEE Int. Conf.
Robot. Autom. (ICRA), 2012, pp. 1410–1415.
[39] D. Göger, N. Gorges, and H. Wörn, “Tactile sensing for an anthropomorphic robotic hand: Hardware and signal processing,” in Proc. IEEE
Int. Conf. Robot. Autom. (ICRA), 2009, pp. 895–901.
[40] H. Yousef, M. Boukallel, and K. Althoefer, “Tactile sensing for
dexterous in-hand manipulation in robotics - a review,” Sensor. Actuat.
A-Phys., vol. 167, no. 2, pp. 171–187, 2011.
[41] J. Tegin, S. Ekvall, D. Kragic, J. Wikander, and B. Iliev,
“Demonstration-based learning and control for automatic grasping,”
Intelligent Service Robotics, vol. 2, no. 1, pp. 23–30, 2009.
[42] X. Song, H. Liu, K. Althoefer, T. Nanayakkara, and L. D. Seneviratne,
“Efficient break-away friction ratio and slip prediction based on haptic
surface exploration,” IEEE Trans. Robot. (T-RO), vol. 30, no. 1, pp.
203–219, 2014.
[43] Y. Bekiroglu, J. Laaksonen, J. A. Jørgensen, V. Kyrki, and D. Kragic,
“Assessing grasp stability based on learning and haptic data,” IEEE
Trans. Robot. (T-RO), vol. 27, no. 3, pp. 616–629, 2011.
[44] M. Li, S. Luo, T. Nanayakkara, L. D. Seneviratne, P. Dasgupta,
and K. Althoefer, “Multi-fingered haptic palpation using pneumatic
feedback actuators,” Sensor. Actuat. A-Phys., vol. 218, pp. 132–141,
2014.
[45] M. Benali-Khoudja, M. Hafez, J.-M. Alexandre, and A. Kheddar,
“Tactile interfaces: a state-of-the-art survey,” in Proc. Int. Symp. Robot.,
vol. 31, 2004, pp. 1–9.
[46] T. Coles, D. Meglan, and N. W. John, “The role of haptics in medical
training simulators: a survey of the state of the art,” IEEE Trans.
Haptics, vol. 4, no. 1, pp. 51–66, 2011.
[47] A. Aggarwal and F. Kirchner, “Object recognition and localization: the
role of tactile sensors,” Sensors, vol. 14, no. 2, pp. 3227–3266, 2014.
[48] C. Liu, L. Sharan, E. H. Adelson, and R. Rosenholtz, “Exploring
features in a Bayesian framework for material recognition,” in Proc.
IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2010, pp. 239–
246.
[49] L. Sharan, C. Liu, R. Rosenholtz, and E. H. Adelson, “Recognizing
materials using perceptually inspired features,” Int. J. Comput. Vision
(IJCV), vol. 103, no. 3, pp. 348–371, 2013.
[50] L. Sun, S. Rogers, G. Aragon-Camarasa, and J. P. Siebert, “Recognising
the clothing categories from free-configuration using gaussian-processbased interactive perception,” in Proc. IEEE Int. Conf. Robot. Auto.
(ICRA), 2016, pp. 2464–2470.
[51] S. J. Lederman and R. L. Klatzky, “Haptic classification of common
objects: Knowledge-driven exploration,” Cognitive Psychology, vol. 22,
no. 4, pp. 421–459, 1990.
[52] W. M. B. Tiest, “Tactual perception of material properties,” Vision
Research, vol. 50, no. 24, pp. 2775–2782, 2010.
[53] S. Luo, L. Zhu, K. Althoefer, and H. Liu, “Knock-knock: Acoustic
object recognition by using stacked denoising autoencoders,” Neurocomputing, 2017.
[54] N. Roy, G. Dudek, and P. Freedman, “Surface sensing and classification
for efficient mobile robot navigation,” in Proc. IEEE Int. Conf. Robot.
Autom. (ICRA), vol. 2, 1996, pp. 1224–1228.
[55] J. Edwards, J. Lawry, J. Rossiter, and C. Melhuish, “Extracting textural
features from tactile sensors,” Bioinspir. Biomim., vol. 3, no. 3, p.
035002, 2008.

PREPRINT SUBMITTED TO ELSEVIER

[56] M. Johnsson and C. Balkenius, “Sense of touch in robots with selforganizing maps,” IEEE Trans. Robot. (T-RO), vol. 27, no. 3, pp. 498–
507, 2011.
[57] N. Jamali and C. Sammut, “Material classification by tactile sensing
using surface textures,” in Proc. IEEE Int. Conf. Robot. Autom. (ICRA),
2010, pp. 2336–2341.
[58] H. Liu, X. Song, J. Bimbo, L. Seneviratne, and K. Althoefer, “Surface
material recognition through haptic exploration using an intelligent
contact sensing finger,” in Proc. IEEE/RSJ Int. Conf. Intell. Robots
Syst. (IROS), 2012, pp. 52–57.
[59] V. A. Ho, T. Araki, M. Makikawa, and S. Hirai, “Experimental
investigation of surface identification ability of a low-profile fabric
tactile sensor,” in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS),
2012, pp. 4497–4504.
[60] P. Dallaire, P. Giguère, D. Émond, and B. Chaib-Draa, “Autonomous
tactile perception: a combined improved sensing and bayesian nonparametric approach,” Robot. Auto. Syst., vol. 62, no. 4, pp. 422–435,
2014.
[61] P. Giguere and G. Dudek, “A simple tactile probe for surface identification by mobile robots,” IEEE Trans. Robot. (T-RO), vol. 27, no. 3,
pp. 534–544, 2011.
[62] J. Sinapov, V. Sukhoy, R. Sahai, and A. Stoytchev, “Vibrotactile
recognition and categorization of surfaces by a humanoid robot,” IEEE
Trans. Robot. (TR-O), vol. 27, no. 3, pp. 488–497, 2011.
[63] J. M. Romano and K. J. Kuchenbecker, “Methods for robotic toolmediated haptic surface recognition,” in Proc. IEEE Haptics Symp.,
2014, pp. 49–56.
[64] M. Kaboli, P. Mittendorfer, V. Hugel, and G. Cheng, “Humanoids
learn object properties from robust tactile feature descriptors via multimodal artificial skin,” in Proc. IEEE-RAS Int. Conf. Human. Robot.
(Humanoids), 2014, pp. 187–192.
[65] M. Tanaka, J. L. Lévêque, H. Tagami, K. Kikuchi, and S. Chonan,
“The haptic finger - a new device for monitoring skin condition,” Skin
Res. Technol., vol. 9, no. 2, pp. 131–136, 2003.
[66] C. M. Oddo, M. Controzzi, L. Beccai, C. Cipriani, and M. C. Carrozza,
“Roughness encoding for discrimination of surfaces in artificial activetouch,” IEEE Trans. Robot. (TR-O), vol. 27, no. 3, pp. 522–533, 2011.
[67] J. A. Fishel and G. E. Loeb, “Bayesian exploration for intelligent
identification of textures,” Front. Neurorobot., 2012.
[68] R. Li and E. H. Adelson, “Sensing and recognizing surface textures
using a GelSight sensor,” in Proc. IEEE Conf. Comput. Vis. Pattern
Recognit. (CVPR), 2013, pp. 1241–1247.
[69] B. Winstone, G. Griffiths, T. Pipe, C. Melhuish, and J. Rossiter, “Tactiptactile fingertip device, texture analysis through optical tracking of skin
features,” in Conf. Biom. Biohy. Sys., 2013, pp. 323–334.
[70] S.-H. Kim, J. Engel, C. Liu, and D. L. Jones, “Texture classification
using a polymer-based MEMS tactile sensor,” J. Micromech. Microeng.,
vol. 15, no. 5, p. 912, 2005.
[71] J. J. Shill, E. G. Collins Jr, E. Coyle, and J. Clark, “Tactile surface
classification for limbed robots using a pressure sensitive robot skin,”
Bioinspir. Biomim., vol. 10, no. 1, p. 016012, 2015.
[72] T. Nanayakkara, A. Jiang, M. d. R. A. Fernández, H. Liu, K. Althoefer,
and J. Bimbo, “Stable grip control on soft objects with time-varying
stiffness,” IEEE Trans. Robot., vol. 32, no. 3, pp. 626–637, 2016.
[73] D. Xu, G. E. Loeb, and J. Fishel, “Tactile identification of objects using
Bayesian exploration,” in Proc. IEEE Int. Conf. Robot. Autom. (ICRA),
2013, pp. 3056–3061.
[74] Z. Su, J. A. Fishel, T. Yamamoto, and G. E. Loeb, “Use of tactile
feedback to control exploratory movements to characterize object
compliance,” Frontiers in Neurorobotics, vol. 6, 2012.
[75] J. Hoelscher, J. Peters, and T. Hermans, “Evaluation of tactile feature
extraction for interactive object recognition,” in Proc. IEEE-RAS Int.
Conf. Human. Robot. (Humanoids), 2015, pp. 310–317.
[76] V. Chu, I. McMahon, L. Riano, C. G. McDonald, Q. He, J. Martinez
Perez-Tejada, M. Arrigo, N. Fitter, J. C. Nappo, T. Darrell, and K. J.
Kuchenbecker, “Using robotic exploratory procedures to learn the
meaning of haptic adjectives,” in Proc. IEEE Int. Conf. Robot. Autom.
(ICRA), 2013, pp. 3048–3055.
[77] J. Windau and W.-M. Shen, “An inertia-based surface identification
system,” in Proc. IEEE Int. Conf. Robot. Autom. (ICRA), 2010, pp.
2330–2335.
[78] W. Yuan, M. A. Srinivasan, and E. H. Adelson, “Estimating object
hardness with a gelsight touch sensor,” in Proc. IEEE/RSJ Int. Conf.
Intell. Robots Syst. (IROS), 2016, pp. 208–215.
[79] W. Yuan, C. Zhu, A. Owens, M. A. Srinivasan, and E. H. Adelson,
“Shape-independent hardness estimation using deep learning and a

14

[80]

[81]
[82]
[83]
[84]
[85]
[86]
[87]
[88]

[89]

[90]

[91]
[92]

[93]
[94]
[95]
[96]
[97]
[98]
[99]
[100]
[101]
[102]
[103]

gelsight tactile sensor,” in Proc. IEEE Int. Conf. Robot. Autom. (ICRA),
2017, pp. 951–958.
S. Decherchi, P. Gastaldo, R. S. Dahiya, M. Valle, and R. Zunino,
“Tactile-data classification of contact materials using computational
intelligence,” IEEE Trans. Robot. (T-RO), vol. 27, no. 3, pp. 635–639,
2011.
A. Drimus, G. Kootstra, A. Bilberg, and D. Kragic, “Design of a
flexible tactile sensor for classification of rigid and deformable objects,”
Robot. Auto. Syst., vol. 62, no. 1, pp. 3–15, 2014.
K. Sangpradit, H. Liu, P. Dasgupta, K. Althoefer, and L. D. Seneviratne,
“Finite-element modeling of soft tissue rolling indentation,” IEEE
Trans. Biom. Eng., vol. 58, no. 12, pp. 3319–3327, 2011.
A. Faragasso, A. Stilli, J. Bimbo, H. A. Wurdemann, and K. Althoefer,
“Multi-axis stiffness sensing device for medical palpation,” in Proc.
IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS), 2015, pp. 2711–2716.
N. Jamali and C. Sammut, “Majority voting: material classification
by tactile sensing using surface texture,” IEEE Trans. Robot. (T-RO),
vol. 27, no. 3, pp. 508–521, 2011.
A. Drimus, G. Kootstra, A. Bilberg, and D. Kragic, “Classification of
rigid and deformable objects using a novel tactile sensor,” in Proc. Int.
Conf. Adv. Robot. (ICAR), 2011, pp. 427–434.
D. G. Lowe, “Object recognition from local scale-invariant features,”
in Proc. IEEE Int. Conf. Compt. Vision (ICCV), 1999, pp. 1150–1157.
P. F. Felzenszwalb and D. P. Huttenlocher, “Pictorial structures for
object recognition,” Int. J. Comput. Vision (IJCV), vol. 61, no. 1, pp.
55–79, 2005.
T. Corradi, P. Hall, and P. Iravani, “Bayesian tactile object recognition:
Learning and recognising objects using a new inexpensive tactile
sensor,” in Proc. IEEE Int. Conf. Robot. Autom. (ICRA), 2015, pp.
3909–3914.
A. Jimenez, A. Soembagijo, D. Reynaerts, H. Van Brussel, R. Ceres,
and J. Pons, “Featureless classification of tactile contacts in a gripper
using neural networks,” Sensor. Actuat. A-Phys., vol. 62, no. 1, pp.
488–491, 1997.
H. Liu, J. Greco, X. Song, J. Bimbo, L. Seneviratne, and K. Althoefer,
“Tactile image based contact shape recognition using neural network,”
in Proc. IEEE Int. Conf. Multi. Fusion Integr. Intell. Syst. (MFI), 2012,
pp. 138–143.
H. Dang, J. Weisz, and P. K. Allen, “Blind grasping: Stable robotic
grasping using tactile feedback and hand kinematics,” in Proc. IEEE
Int. Conf. Robot. Autom. (ICRA), 2011, pp. 5917–5922.
U. Martinez-Hernandez, T. J. Dodd, L. Natale, G. Metta, T. J. Prescott,
and N. F. Lepora, “Active contour following to explore object shape
with robot touch,” in Proc. IEEE World Haptics Conf. (WHC), 2013,
pp. 341–346.
M. Schöpfer, M. Pardowitz, and H. Ritter, “Using entropy for dimension reduction of tactile data,” in Proc. Int. Conf. Adv. Robot. (ICAR),
2009, pp. 1–6.
D. S. Tawil, D. Rye, and M. Velonaki, “Interpretation of the modality
of touch on an artificial arm covered with an eit-based sensitive skin,”
Int. J. Robot. Res. (IJRR), vol. 31, no. 13, pp. 1627–1641, 2012.
D. S. Tawil, D. Rye, and M. Velonaki, “Touch modality interpretation
for an EIT-based sensitive skin,” in Proc. IEEE Int. Conf. Robot. Autom.
(ICRA), 2011, pp. 3770–3776.
S. Chitta, J. Sturm, M. Piccoli, and W. Burgard, “Tactile sensing for
mobile manipulation,” IEEE Trans. Robot. (T-RO), vol. 27, no. 3, pp.
558–568, 2011.
R. A. Russell, “Object recognition by a smarttactile sensor,” in Proc.
Austr. Conf. Robot. Autom., 2000, pp. 93–98.
A. J. Schmid, N. Gorges, D. Goger, and H. Worn, “Opening a door
with a humanoid robot using multi-sensory tactile feedback,” in Proc.
IEEE Int. Conf. Robot. Autom. (ICRA), 2008, pp. 285–291.
A. Khasnobish, G. Singh, A. Jati, A. Konar, and D. Tibarewala,
“Object-shape recognition and 3D reconstruction from tactile sensor
images,” Med. Biol. Eng. Comput., vol. 52, no. 4, pp. 353–362, 2014.
D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” Int. J. Comput. Vision (IJCV), vol. 60, no. 2, pp. 91–110,
2004.
S. Luo, W. Mou, M. Li, K. Althoefer, and H. Liu, “Rotation and
translation invariant object recognition with a tactile sensor,” in Proc.
IEEE Sensors Conf., 2014, pp. 1030–1033.
S. Luo, W. Mou, K. Althoefer, and H. Liu, “Novel Tactile-SIFT
descriptor for object shape recognition,” IEEE Sensors J., vol. 15, no. 9,
pp. 5001–5009, 2015.
S. Luo, X. Liu, K. Althoefer, and H. Liu, “Tactile object recognition
with semi-supervised learning,” in Proc. Int. Conf. Intell. Robot. and
Appl. (ICIRA), 2015, pp. 15–26.

PREPRINT SUBMITTED TO ELSEVIER

[104] K. Mikolajczyk and C. Schmid, “A performance evaluation of local
descriptors,” IEEE Trans. Pattern Anal. Mach. Intell. (T–PAMI), vol. 27,
no. 10, pp. 1615–1630, 2005.
[105] S. Gauglitz, T. Höllerer, and M. Turk, “Evaluation of interest point
detectors and feature descriptors for visual tracking,” Int. J. Comput.
Vision (IJCV), vol. 94, no. 3, pp. 335–360, 2011.
[106] S. Belongie, J. Malik, and J. Puzicha, “Shape matching and object
recognition using shape contexts,” IEEE Trans. Pattern Anal. Mach.
Intell. (T–PAMI), vol. 24, no. 4, pp. 509–522, 2002.
[107] H. Bay, T. Tuytelaars, and L. Van Gool, “Surf: Speeded up robust
features,” in Proc. Eur. Conf. Comput. Vis. (ECCV), 2006, pp. 404–
417.
[108] F. Tombari, S. Salti, and L. Di Stefano, “Unique signatures of histograms for local surface description,” in Proc. Eur. Conf. Comput.
Vis. (ECCV), 2010, pp. 356–369.
[109] V. A. Ho, T. Nagatani, A. Noda, and S. Hirai, “What can be inferred
from a tactile arrayed sensor in autonomous in-hand manipulation?” in
Proc. IEEE Int. Conf. Autom. Sci. and Eng. (CASE), 2012, pp. 461–468.
[110] Z. Ji, F. Amirabdollahian, D. Polani, and K. Dautenhahn, “Histogram
based classification of tactile patterns on periodically distributed skin
sensors for a humanoid robot,” in Proc. IEEE Int. Symp. Robot Hum.
Intert. Commun. (RO-MAN), 2011, pp. 433–440.
[111] C. Schürmann, R. Koiva, R. Haschke, and H. Ritter, “A modular highspeed tactile sensor for human manipulation research,” in Proc. IEEE
World Haptics Conf. (WHC), 2011, pp. 339–344.
[112] R. D. P. Wong, R. B. Hellman, and V. J. Santos, “Spatial asymmetry
in tactile sensor skin deformation aids perception of edge orientation
during haptic exploration,” IEEE Trans. Haptics, vol. 7, no. 2, pp. 191–
202, 2014.
[113] Y. Bekiroglu, R. Detry, and D. Kragic, “Learning tactile characterizations of object-and pose-specific grasps,” in Proc. IEEE/RSJ Int. Conf.
Intell. Robots Syst. (IROS), 2011, pp. 1554–1560.
[114] Y. Bekiroglu, D. Song, L. Wang, and D. Kragic, “A probabilistic
framework for task-oriented grasp stability assessment,” in Proc. IEEE
Int. Conf. Robot. Autom. (ICRA), 2013, pp. 3040–3047.
[115] Q. Li, C. Schürmann, R. Haschke, and H. Ritter, “A control framework
for tactile servoing,” in Proc. Robot.: Sci. Syst. (RSS), 2013.
[116] R. Li, R. Platt, W. Yuan, A. ten Pas, N. Roscup, M. A. Srinivasan,
and E. Adelson, “Localization and manipulation of small parts using
GelSight tactile sensing,” in Proc. IEEE/RSJ Int. Conf. Intell. Robots
Syst. (IROS), 2014, pp. 3988–3993.
[117] N. Gorges, S. E. Navarro, D. Göger, and H. Wörn, “Haptic object
recognition using passive joints and haptic key features,” in Proc. IEEE
Int. Conf. Robot. Autom. (ICRA), 2010, pp. 2349–2355.
[118] Y. H. Liu, Y. T. Hsiao, W. T. Cheng, Y. C. Liu, and J. Y. Su, “Lowresolution tactile image recognition for automated robotic assembly
using kernel PCA-based feature fusion and multiple kernel learningbased support vector machine,” Math. Probl. Eng., 2014.
[119] M. Li, Y. Bekiroglu, D. Kragic, and A. Billard, “Learning of grasp
adaptation through experience and tactile sensing,” in Proc. IEEE/RSJ
Int. Conf. Intell. Robots Syst. (IROS), 2014, pp. 3339–3346.
[120] M. Schöpfer, H. Ritter, and G. Heidemann, “Acquisition and application
of a tactile database,” in Proc. IEEE Int. Conf. Robot. Autom. (ICRA),
2007, pp. 1517–1522.
[121] J. Bimbo, S. Luo, K. Althoefer, and H. Liu, “In-Hand Object Pose
Estimation Using Covariance-Based Tactile To Geometry Matching,”
IEEE Robot. Auto. Lett. (RA-L), vol. 1, no. 1, pp. 570–577, 2016.
[122] M. Madry, L. Bo, D. Kragic, and D. Fox, “ST-HMP: Unsupervised
spatio-temporal feature learning for tactile data,” in Proc. IEEE Int.
Conf. Robot. Autom. (ICRA), 2014, pp. 2262–2269.
[123] H. Soh, Y. Su, and Y. Demiris, “Online spatio-temporal gaussian
process experts with application to tactile classification,” in Proc.
IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS), 2012, pp. 4489–4496.
[124] H. Soh and Y. Demiris, “Incrementally Learning Objects by Touch:
Online Discriminative and Generative Models for Tactile-based Recognition,” IEEE Trans. Haptics, vol. 7, no. 4, pp. 512–525, 2014.
[125] A. Molchanov, O. Kroemer, Z. Su, and G. S. Sukhatme, “Contact localization on grasped objects using tactile sensing,” in Proc. IEEE/RSJ
Int. Conf. Intell. Robots Syst. (IROS), 2016, pp. 216–222.
[126] A. Schmitz, Y. Bansho, K. Noda, H. Iwata, T. Ogata, and S. Sugano,
“Tactile Object Recognition using Deep Learning and Dropout,” in
Proc. IEEE-RAS Int. Conf. Human. Robot. (Humanoids), 2014, pp.
1044–1050.
[127] L. Cao, R. Kotagiri, F. Sun, H. Li, W. Huang, and Z. M. M. Aye,
“Efficient spatio-temporal tactile object recognition with randomized
tiling convolutional networks in a hierarchical fusion strategy,” in Proc.
AAAI National Conf. Artif. Intell. (AAAI), 2016, pp. 3337–3345.

15

[128] H. Liu, D. Guo, and F. Sun, “Object recognition using tactile measurements: Kernel sparse coding methods,” IEEE Trans. Instrum. Meas.,
vol. 65, no. 3, pp. 656–665, 2016.
[129] Y. Bengio, A. Courville, and P. Vincent, “Representation learning: A
review and new perspectives,” IEEE Trans. Pattern Anal. Mach. Intell.
(T–PAMI), vol. 35, no. 8, pp. 1798–1828, 2013.
[130] E. Hyttinen, D. Kragic, and R. Detry, “Learning the tactile signatures
of prototypical object parts for robust part-based grasping of novel
objects,” in Proc. IEEE Int. Conf. Robot. Autom. (ICRA), 2015, pp.
4927–4932.
[131] J. A. Stork, C. H. Ek, Y. Bekiroglu, and D. Kragic, “Learning Predictive
State Representation for in-hand manipulation,” in Proc. IEEE Int.
Conf. Robot. Autom. (ICRA), 2015, pp. 3207–3214.
[132] W. E. L. Grimson and T. Lozano-Pérez, “Model-based recognition and
localization from sparse range or tactile data,” Int. J. Robot. Res. (IJRR),
vol. 3, no. 3, pp. 3–35, 1984.
[133] P. K. Allen and K. S. Roberts, “Haptic object recognition using a
multi-fingered dextrous hand,” in Proc. IEEE Int. Conf. Robot. Autom.
(ICRA), 1989, pp. 342–347.
[134] M. Charlebois, K. Gupta, and S. Payandeh, “Shape description of
curved surfaces from contact sensing using surface normals,” Int. J.
Robot. Res. (IJRR), vol. 18, no. 8, pp. 779–787, 1999.
[135] A. M. Okamura and M. R. Cutkosky, “Feature detection for haptic
exploration with robotic fingers,” Int. J. Robot. Res. (IJRR), vol. 20,
no. 12, pp. 925–938, 2001.
[136] S. Casselli, C. Magnanini, and F. Zanichelli, “On the robustness of
haptic object recognition based on polyhedral shape representations,”
in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS), 1995, pp.
200–206.
[137] R. S. Fearing and T. O. Binford, “Using a cylindrical tactile sensor for
determining curvature,” IEEE Trans. Robot. Autom., vol. 7, no. 6, pp.
806–817, 1991.
[138] R. Ibrayev and Y. B. Jia, “Semidifferential invariants for tactile recognition of algebraic curves,” Int. J. Robot. Res. (IJRR), vol. 24, no. 11,
pp. 951–969, 2005.
[139] Y. B. Jia, L. Mi, and J. Tian, “Surface patch reconstruction via curve
sampling,” in Proc. IEEE Int. Conf. Robot. Autom. (ICRA), 2006, pp.
1371–1377.
[140] Y. B. Jia and J. Tian, “Surface patch reconstruction from onedimensional tactile data,” , IEEE Trans. Autom. Sci. Eng., vol. 7, no. 2,
pp. 400–407, 2010.
[141] I. Abraham, A. Prabhakar, M. J. Hartmann, and T. D. Murphey,
“Ergodic exploration using binary sensing for nonparametric shape
estimation,” IEEE Robot. Auto. Lett. (RA-L), vol. 2, no. 2, pp. 827–834,
2017.
[142] Z. Pezzementi, C. Reyda, and G. D. Hager, “Object mapping, recognition, and localization from tactile geometry,” in Proc. IEEE Int. Conf.
Robot. Autom. (ICRA), 2011, pp. 5942–5948.
[143] M. M. Zhang, M. D. Kennedy, M. A. Hsieh, and K. Daniilidis, “A
triangle histogram for object classification by tactile sensing,” in Proc.
IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS), 2016, pp. 4931–4938.
[144] M. Meier, M. Schöpfer, R. Haschke, and H. Ritter, “A probabilistic
approach to tactile shape reconstruction,” IEEE Trans. Robot. (T-RO),
vol. 27, no. 3, pp. 630–635, 2011.
[145] A. Aggarwal, P. Kampmann, J. Lemburg, and F. Kirchner, “Haptic
Object Recognition in Underwater and Deep-sea Environments,” J.
Field Robot., vol. 32, no. 1, pp. 167–185, 2015.
[146] E. Nowak, F. Jurie, and B. Triggs, “Sampling strategies for bag-offeatures image classification,” in Proc. Eur. Conf. Comput. Vis. (ECCV),
2006, pp. 490–503.
[147] D. Prattichizzo, F. Chinello, C. Pacchierotti, and M. Malvezzi, “Towards wearability in fingertip haptics: A 3-dof wearable device for
cutaneous force feedback,” IEEE Trans. Haptics, vol. 6, no. 4, pp.
506–516, 2013.
[148] C. Pacchierotti, A. Tirmizi, G. Bianchini, and D. Prattichizzo, “Improving transparency in passive teleoperation by combining cutaneous
and kinesthetic force feedback,” in Proc. IEEE/RSJ Int. Conf. Intell.
Robots Syst. (IROS), 2013, pp. 4958–4963.
[149] W. McMath, S. Yeung, E. Petriu, and N. Trif, “Tactile sensor for
geometric profile perception,” in Proc. IEEE Int. Conf. Indust. Elect.
Cont. Inst., 1991, pp. 893–897.
[150] E. M. Petriu, W. S. McMath, S. S. Yeung, and N. Trif, “Active tactile
perception of object surface geometric profiles,” IEEE Trans. Inst.
Meas., vol. 41, no. 1, pp. 87–92, 1992.
[151] M. Johnsson and C. Balkenius, “Neural network models of haptic shape
perception,” Robot. Auto. Syst., vol. 55, no. 9, pp. 720–727, 2007.

PREPRINT SUBMITTED TO ELSEVIER

[152] S. E. Navarro, N. Gorges, H. Worn, J. Schill, T. Asfour, and R. Dillmann, “Haptic object recognition for multi-fingered robot hands,” in
Proc. IEEE Haptics Symp., 2012, pp. 497–502.
[153] A. Spiers, M. Liarokapis, B. Calli, and A. Dollar, “Single-Grasp Object
Classification and Feature Extraction with Simple Robot Hands and
Tactile Sensors,” IEEE Trans. Haptics, vol. 9, no. 2, pp. 207–220,
2016.
[154] S. Luo, W. Mou, K. Althoefer, and H. Liu, “Iterative closest labeled
point for tactile object shape recognition,” in Proc. IEEE/RSJ Int. Conf.
Intell. Robots Syst. (IROS), 2016, pp. 3137–3142.
[155] K. Hertkorn, M. A. Roa, and C. Borst, “Planning in-hand object
manipulation with multifingered hands considering task constraints,”
in Proc. IEEE Int. Conf. Robot. Autom. (ICRA), 2013, pp. 617–624.
[156] D. M. Siegel, “Finding the pose of an object in a hand,” in Proc. IEEE
Int. Conf. Robot. Autom. (ICRA), 1991, pp. 406–411.
[157] K. Gadeyne, T. Lefebvre, and H. Bruyninckx, “Bayesian hybrid modelstate estimation applied to simultaneous contact formation recognition
and geometrical parameter estimation,” Int. J. Robot. Res. (IJRR),
vol. 24, no. 8, pp. 615–630, 2005.
[158] M. Montemerlo and S. Thrun, “Simultaneous localization and mapping
with unknown data association using FastSLAM,” in Proc. IEEE Int.
Conf. Robot. Autom. (ICRA), vol. 2, 2003, pp. 1985–1991.
[159] M. Schaeffer, A. M. Okamura et al., “Methods for intelligent localization and mapping during haptic exploration,” in Proc. IEEE Int. Conf.
Syst., Man, and Cyber. (SMC), 2003, pp. 3438–3445.
[160] C. Corcoran and R. Platt Jr, “A measurement model for tracking handobject state during dexterous manipulation,” in Proc. IEEE Int. Conf.
Robot. Autom. (ICRA), 2010, pp. 4302–4308.
[161] R. Platt Jr, F. Permenter, and J. Pfeiffer, “Using bayesian filtering to
localize flexible materials during manipulation,” IEEE Trans. Robot.
(T-RO), vol. 27, no. 3, pp. 586–598, 2011.
[162] A. Petrovskaya, O. Khatib, S. Thrun, and A. Y. Ng, “Bayesian estimation for autonomous object manipulation based on tactile sensors,” in
Proc. IEEE Int. Conf. Robot. Autom. (ICRA), 2006, pp. 707–714.
[163] A. Petrovskaya, O. Khatib, S. Thrun, and A. Y. Ng, “Touch based
perception for object manipulation,” in Proc. Workshop Robot Manip.,
Robot.: Sci. Syst. (RSS), 2007, pp. 2–7.
[164] A. Petrovskaya and O. Khatib, “Global localization of objects via
touch,” IEEE Trans. Robot. (T-RO), vol. 27, no. 3, pp. 569–585, 2011.
[165] G. Vezzani, U. Pattacini, G. Battistelli, L. Chisci, and L. Natale,
“Memory unscented particle filter for 6–DOF tactile localization,” IEEE
Trans. Robot. (T-RO), 2017.
[166] K. T. Yu, J. Leonard, and A. Rodriguez, “Shape and pose recovery
from planar pushing,” in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst.
(IROS), 2015, pp. 1208–1215.
[167] A. J. Davison, I. D. Reid, N. D. Molton, and O. Stasse, “MonoSLAM:
Real-time single camera SLAM,” IEEE Trans. Pattern Anal. Mach.
Intell. (T–PAMI), vol. 29, no. 6, pp. 1052–1067, 2007.
[168] T. Bhattacharjee, A. A. Shenoi, D. Park, J. M. Rehg, and C. C. Kemp,
“Combining tactile sensing and vision for rapid haptic mapping,” in
Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS), 2015, pp. 1200–
1207.
[169] S. Luo, W. Mou, K. Althoefer, and H. Liu, “Localizing the object
contact through matching tactile features with visual map,” in Proc.
IEEE Int. Conf. Robot. Autom. (ICRA), 2015, pp. 3903–3908.
[170] J. Bimbo, P. Kormushev, K. Althoefer, and H. Liu, “Global estimation
of an objects pose using tactile sensing,” Adv. Robotics, vol. 29, no. 5,
pp. 363–374, 2015.
[171] M. C. Koval, N. S. Pollard, and S. S. Srinivasa, “Pose estimation for
planar contact manipulation with manifold particle filters,” Int. J. Robot.
Res. (IJRR), vol. 34, no. 7, pp. 922–945, 2015.
[172] P. Hebert, N. Hudson, J. Ma, and J. Burdick, “Fusion of stereo
vision, force-torque, and joint sensors for estimation of in-hand object
location,” in Proc. IEEE Int. Conf. Robot. Autom. (ICRA), 2011, pp.
5935–5941.
[173] K. Honda, T. Hasegawa, T. Kiriki, and T. Matsuoka, “Real-time pose
estimation of an object manipulated by multi-fingered hand using 3d
stereo vision and tactile sensing,” in Proc. IEEE/RSJ Int. Conf. Intell.
Robots Syst. (IROS), vol. 3, 1998, pp. 1814–1819.
[174] J. Bimbo, L. D. Seneviratne, K. Althoefer, and H. Liu, “Combining
touch and vision for the estimation of an object’s pose during manipulation,” in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS), 2013,
pp. 4021–4026.
[175] S. Lacey, C. Campbell, and K. Sathian, “Vision and touch: multiple or
multisensory representations of objects?” Perception, vol. 36, no. 10,
pp. 1513–1522, 2007.

16

[176] J. Felip, A. Morales, and T. Asfour, “Multi–sensor and prediction fusion
for contact detection and localization,” in Proc. IEEE-RAS Int. Conf.
Human. Robot. (Humanoids), 2014, pp. 601–607.
[177] R. C. Luo, C.-C. Yih, and K. L. Su, “Multisensor fusion and integration:
approaches, applications, and future research directions,” IEEE Sensors
J., vol. 2, no. 2, pp. 107–119, 2002.
[178] M. Prats, P. J. Sanz, and A. P. del Pobil, “Vision–tactile–force integration and robot physical interaction,” in Proc. IEEE Int. Conf. Robot.
Autom. (ICRA), 2009, pp. 3975–3980.
[179] R. C. Luo and M. G. Kay, “Multisensor integration and fusion in
intelligent systems,” IEEE Trans. Sys., Man Cyber., vol. 19, no. 5,
pp. 901–931, 1989.
[180] P. Allen, “Surface descriptions from vision and touch,” in Proc. IEEE
Int. Conf. Robot. Autom. (ICRA), 1984, pp. 394–397.
[181] M. Bjorkman, Y. Bekiroglu, V. Hogman, and D. Kragic, “Enhancing
visual perception of shape through tactile glances,” in Proc. IEEE/RSJ
Int. Conf. Intell. Robots Syst. (IROS), 2013, pp. 3180–3186.
[182] J. Ilonen, J. Bohg, and V. Kyrki, “Fusing visual and tactile sensing for
3-D object reconstruction while grasping,” in Proc. IEEE Int. Conf.
Robot. Autom. (ICRA), 2013, pp. 3547–3554.
[183] J. Sinapov, T. Bergquist, C. Schenck, U. Ohiri, S. Griffith, and
A. Stoytchev, “Interactive object recognition using proprioceptive and
auditory feedback,” Int. J. Robot. Res. (IJRR), vol. 30, no. 10, pp.
1250–1262, 2011.
[184] T. Araki, T. Nakamura, T. Nagai, K. Funakoshi, M. Nakano, and
N. Iwahashi, “Online object categorization using multimodal information autonomously acquired by a mobile robot,” Adv. Robotics, vol. 26,
no. 17, pp. 1995–2020, 2012.
[185] P. K. Allen, “Integrating vision and touch for object recognition tasks,”
Int. J. Robot. Res. (IJRR), vol. 7, no. 6, pp. 15–33, 1988.
[186] P. K. Allen, A. T. Miller, P. Y. Oh, and B. S. Leibowitz, “Integration of
vision, force and tactile sensing for grasping,” in Int. J. Intell. Mach.,
vol. 4, 1999, pp. 129–149.
[187] S. Dragiev, M. Toussaint, and M. Gienger, “Gaussian process implicit
surfaces for shape estimation and grasping,” in Proc. IEEE Int. Conf.
Robot. Autom. (ICRA), 2011, pp. 2845–2850.
[188] P. Guler, Y. Bekiroglu, X. Gratal, K. Pauwels, and D. Kragic, “What’s
in the container? classifying object contents from vision and touch,”
in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS), 2014, pp.
3961–3968.
[189] M. Prats, P. J. Sanz, and A. P. del Pobil, “Reliable non-prehensile door
opening through the combination of vision, tactile and force feedback,”
Auton. Robot., vol. 29, no. 2, pp. 201–218, 2010.
[190] G. Izatt, G. Mirano, E. Adelson, and R. Tedrake, “Tracking objects
with point clouds from vision and touch,” in Proc. IEEE Int. Conf.
Robot. Autom. (ICRA), 2017, pp. 4000–4007.
[191] A. Zangaladze, C. M. Epstein, S. T. Grafton, and K. Sathian, “Involvement of visual cortex in tactile discrimination of orientation,” Nature,
vol. 401, no. 6753, pp. 587–590, 1999.
[192] A. Amedi, G. Jacobson, T. Hendler, R. Malach, and E. Zohary,
“Convergence of visual and tactile shape processing in the human
lateral occipital complex,” Cereb. Cortex, vol. 12, no. 11, pp. 1202–
1212, 2002.
[193] M. Sanchez-Fibla, A. Duff, and P. F. Verschure, “A sensorimotor
account of visual and tactile integration for object categorization and
grasping,” in Proc. IEEE Int. Conf. Robot. Autom. (ICRA), 2013, pp.
107–112.
[194] Y. Gao, L. A. Hendricks, K. J. Kuchenbecker, and T. Darrell, “Deep
learning for tactile understanding from visual and haptic data,” in Proc.
IEEE Int. Conf. Robot. Autom. (ICRA), 2016, pp. 536–543.
[195] W. Yuan, S. Wang, S. Dong, and E. Adelson, “Connecting look and
feel: Associating the visual and tactile properties of physical materials,”
in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2017.
[196] W. W. Lee, S. L. Kukreja, and N. V. Thakor, “A kilohertz kilotaxel
tactile sensor array for investigating spatiotemporal features in neuromorphic touch,” in IEEE Biom. Circ. Sys. Conf. (BioCAS), 2015, pp.
1–4.
[197] C. Bartolozzi, R. Benosman, K. Boahen, G. Cauwenberghs, T. Delbrck,
G. Indiveri, S.-C. Liu, S. Furber, N. Imam, B. Linares-Barranco,
T. Serrano-Gotarredona, K. Meier, C. Posch, and M. Valle, “Neuromorphic systems,” in Wiley Encyclopedia of Electrical and Electronics
Engineering, 2016.
[198] J. S. Son, E. A. Monteverde, and R. D. Howe, “A tactile sensor for
localizing transient events in manipulation,” in Proc. IEEE Int. Conf.
Robot. Autom. (ICRA), 1994, pp. 471–476.

PREPRINT SUBMITTED TO ELSEVIER

Shan Luo is currently a Postdoctoral Research
Fellow at Brigham and Women’s Hospital, Harvard
Medical School. Previous to this position, he was
a Research Fellow at University of Leeds and a
Visiting Scientist at Computer Science and Artificial
Intelligence Laboratory (CSAIL), Massachusetts Institute of Technology (MIT). He received the B.Eng.
degree in Automatic Control from China University
of Petroleum, Qingdao, China, in 2012. He was
awarded the Ph.D. degree in Robotics from King’s
College London, UK, in 2016. His research interests
include tactile sensing, object recognition and computer vision.

Joao Bimbo is currently a Postdoctoral Researcher
at the Istituto Italiano di Tecnologia. He received
his MSc. degree in Electrical Engineering from the
University of Coimbra, Portugal, in 2011. He was
awarded the Ph.D. degree in Robotics from King’s
College London, UK, in 2016. His research interests
include tactile sensing and teleoperation.

17

Ravinder Dahiya is Professor and EPSRC Fellow in
the School of Engineering at University of Glasgow,
UK. He is Director of Electronics Systems Design
Centre in University of Glasgow and leader of Bendable Electronics and Sensing Technologies (BEST)
group. He completed Ph.D. at Italian Institute of
Technology, Genoa (Italy). In past, he worked at
Delhi University (India), Italian Institute of Technology, Genoa (Italy), Fondazione Bruno Kessler,
Trento (Italy), and University of Cambridge (UK).
His multidisciplinary research interests include
Flexible and Printable Electronics, Electronic Skin, Tactile Sensing, robotics,
and wearable electronics. He has published more than 160 research articles,
4 books (including 3 in various stage of publication) and 9 patents (including
submitted). He has worked on and led many international projects.
He is Distinguished Lecturer of IEEE Sensors Council and senior member
of IEEE. Currently he is serving on the Editorial Boards of Scientific Reports
(Nature Pub. Group), IEEE Transactions on Robotics and IEEE Sensors
Journal and has been guest editor of 4 Special Journal Issues. He is member
of the AdCom of IEEE Sensors Council and is the founding chair of the IEEE
UKRI sensors council chapter. He was General Chair of IEEE PRIME 2015
and is the Technical Program Chair (TPC) of IEEE Sensors 2017.
Prof. Dahiya holds prestigious EPSRC Fellowship and also received Marie
Curie Fellowship and Japanese Monbusho Fellowship. He was awarded with
the University Gold Medal and received best paper awards 2 times and
another 2 second best paper awards (as co-author) in the IEEE international
conferences. He has received numerous awards including the 2016 IEEE
Sensors Council Technical Achievement award and 2016 Microelectronics
Young Investigator Award. Dr. Dahiya was among Scottish 40under40 in 2016.

Hongbin Liu is a Senior Lecturer (Associate Professor) in the Department of Informatics, King’s
College London (KCL) where he is directing the
Haptic Mechatronics and Medical Robotics Laboratory (HaMMeR) within the Centre for Robotics
Research (CoRe). Dr. Liu obtained his BEng in 2005
from Northwestern Polytechnical University, China,
MSc and PhD in 2006 and 2010 respectively, both
from KCL. He is a Technical Committee Member
of IEEE EMBS BioRobotics. He has published over
100 peer-reviewed publications at top international
robotic journals and conferences. His research lies in creating the artificial
haptic perception for robots with soft and compliant structures, and making
use of haptic sensation to enable the robot to effectively physically interact
with complex and changing environment. His research has been funded by
EPSRC, Innovate UK, NHS Trust and EU Commissions.

