Hand Gesture Recognition with Leap Motion

arXiv:1711.04293v1 [cs.CV] 12 Nov 2017

Youchen Du, Shenglan Liu, Lin Feng, Menghui Chen, Jie Wu
Dalian University of Technology

Abstract
The recent introduction of depth cameras like Leap
Motion Controller allows researchers to exploit the
depth information to recognize hand gesture more
robustly. This paper proposes a novel hand gesture
recognition system with Leap Motion Controller. A
series of features are extracted from Leap Motion
tracking data, we feed these features along with HOG
feature extracted from sensor images into a multiclass SVM classifier to recognize performed gesture,
dimension reduction and feature weighted fusion are
also discussed. Our results show that our model is
much more accurate than previous work.
Index Terms–Gesture Recognition, Leap Motion
Controller, SVM, PCA, Feature Fusion, Depth

1

INTRODUCTION

In recent years, with the enormous development in
the field of machine learning, problems such as understanding human voice, language, movement, posture become more and more popular, hand gesture
recognition as one of the these fields has attracted
many researchers’s interest[1]. Hand is an important part of the human body, as a way to supplement
the human language, gestures play an important role
in daily life, in the fields of human-computer interaction, robotics, sign-language, how to recognize a
hand gesture is one of the core issues[2][3][4]. In previous work, Orientation Histograms have been used
to recognize hand gesture[5], a variant of EMD also
have been used to finish this task[6]. Recently, a
bunch of depth cameras such as Time-of-Flight cameras and Microsoft Kinecthave been marketed one after another, the use of depth features has been added
to the gesture recognition based on low dimentional
feature extraction[7]. A volumetric shape descriptor
have been used to achieved robust pose recognition
in real time[8], adding features like distance, eleva-

tion, curvature based on 3D information on the hand
shape and finger posture contained in depth data have
also improved accuracy[9]. Recognize hand gesture
through contour have also been explored[10]. Use finger segmentation to recognize hand gesture have been
tested[11]. Use HOG feature and SVM to recognize
hand gesture have also been proposed[12].
The Leap Motion Controller is a consumer-oriented
tool for gesture recognition and finger positioning developed by Leap Motion. Unlike products like the Microsoft Kinect, it is based on binocular visual depth
and provides data on fine-grained locations such as
hands and knuckles. Due to the different design concepts, it can only work normally under close conditions, but it has a good performance on data accuracy with an accuracy of 0.2mm[13]. There have been
many researches try to recognize hand gesture Leap
Motion Controller[14][15]. Combining Leap Motion
and Kinect for hand gesture recognition have been
proposed and achieved a good accuracy[16].
Our contributions are as follows:
1. We propose a Leap Motion Controller hand gesture dataset, which contains 13 subjects and 10
gestures, each gesture by each subject is repeated
20 times, thus we have 2600 samples in total.
2. We use Leap Motion only. Based on the Leap
Motion Controller part of [1], we propose a new
feature called Fingertips Tip distance(T). With
the use of the new feature, we observed considerable accuracy improvements on both of the data
provided by [1] and our data.
3. We extract the HOG feature of Leap Motion
Controller sensor images, HOG feature significantly improved gesture accuracy.
4. We explore different weight coefficients to combine the features of raw tracking data and the
HOG feature of raw sensor images, by applying
dimension reduction with principal component

Page 1

Figure 1: System architecture

Figure 2: Capture setup

analysis, we finally get best accuracy of 99.62%
on our dataset.
This paper is organized in this way: In Section II,
we give a brief introduction of our model architecture, methods and our dataset. In Section III, we
show the new Fingertips Tip Distance feature on the
basis of previous work. In Section VI, we present
the HOG feature extracted from sensor images from
Leap Motion Controller. In Section V, we analyze and
compare the performance of the new feature with the
work presented by Margin et al, verifies the performance of the HOG feature alone, and evaluates the
performance after applying PCA. In Section VI, we
put forward the conclusion of this paper and thoughts
on the following work.

Figure 3: Gestures in dataset

2.2

2

OVERVIEW

In this section, we describe the model architecture
we use and the way data is handled(Section 2.1),
and how we collect our dataset by Leap Motion Controller(Section 2.2).

2.1

SYSTEM ARCHITECTURE

Fig 1 shows in detail the recognition model we
designed. The tracking data and sensor images of
the gesture are captured simultaneously, for tracking
data, we extract a series of related features, for sensor images, we extract the HOG feature, with some
different weight coefficients, we apply feature fusion
and dimension reduction by PCA, finally put these
features into a One-vs-One multi-class SVM to classify hand gesture.

HAND GESTURE DATASET

In order to evaluate the effect of the new feature
and test the HOG feature of the sensor images, we
propose a new dataset, the setup is shown in Fig
2. The dataset contains a total of 10 gestures(Fig
3) performed by 13 individuals, each gesture is repeated 20 times, so the dataset contains a total of
2600 samples. The tracking data and sensor images
are captured simultaneously, and each individual is
told to perform gestures within Leap Motion Controller’s valid visual range, allowing translation and
rotation, no other prior knowledges.

3

FEATURE EXTRACTION
FROM TRACKING DATA

Margin et al. proposed features like Fingertips
angle(A), Fingertips distance(D), Fingertips eleva-

Page 2

Figure 4: Positions and Directions of fingertips

Figure 5: The model of thumb from Leap Motion

tion(E) from tracking data of Leap Motion Controller,
as defined below:
Ai = 6 (Fiπ − C, h)
kFi − Ck
S

(2)

sgn((Fi − Fiπ ) · n) kFi − Fiπ k
S

(3)

Di =
Ei =

(1)

Where S = Fmiddle − C, C is the coordinates of
palm center in 3D space relative to the Controller, h
is the vector of directions from the palm center to the
fingers, n is the direction vector pointing perpendicular to the palm plane, as shown in Fig 4 . Fi is the
coordinate of the detected fingertip, i = 1, · · · , 5, the
data is not associated with any specified finger. Fiπ
is the projection of Fi on the plane determined by n.
Fmiddle is the coordinates of the middle finger after
associate F with fingers. Ai will be used to associate
data to a specified finger, note that invalid Ai will be
set to 0 while the rest of the Ai will be scaled to [0.5,
1], please refer to [1] for more details.
The previous study only considered the relationship between fingers and palm, and did not consider the possible relationship among fingers(Fig 5).
Therefore, we did some research for this part of work.
By calculating the distance between fingertips and arranging them in ascending order of distance, we get
a new feature we called Fingertips Tip Distance(T),
as defined below:
Tij =

kFi − Fj k
, 1 ≤ i 6= j ≤ 5
S

(4)

After adding this feature, the performance has been
further improved, the specific results will be given in
Section V.

Figure 6: Raw images from Leap Motion Controller

4
4.1

FEATURE EXTRACTION
FROM SENSOR IMAGES
Sensor images preprocessing

Barrel distortion is introduced due to Leap Motion
Controller’s hardware(Fig 6), in order to get realistic
images we use a official method from Leap Motion to
use bilinear interpolation to correct distorted images.
We use threshold filtering for the corrected image,
and after doing so, the image will be binarized, retaining the area of the hand and removing the non-hand
area as much as possible, as show in Fig 7.

4.2

Histogram of Oriented Gradient

The HOG feature is a feature descriptor used for
object detection in computer vision and image processing. Its essence is the statistics of image gradient
information. In this paper, we use HOG feature to

Page 3

Features Combination
D+E+T
A+E+T
A+D+E
A+D+T
A+D+E+T

Dataset from Marin et al.
72.33%
79.31%
79.80%
80.81%
81.08%

Table 1: Different features combination performance
on different datasets

Figure 7: Binarized images
extract the feature information about gestures in binarized undistorted sensor images.

4.3

Feature fusion

For features like HOG from sensor images and
ADET features from tracking data, we try to weight
these features and feed into SVM to perform classification. we have tried to magnify the HOG feature
differences by a factor of 1-9 and compare these results.

5
5.1

EXPERIMENTS AND RESULTS
SVM architecture

We use the One-vs-One strategy for multi-class
RBF SVM to classify 10 classes, for each class pair
there is a SVM, so result in a total of 10∗(10−1)/2 =
45 classifiers, the final classification result based on
votes received. For hyper-parameters like (C, γ), we
use the grid search method on 80% of the samples
with 10-fold cross-validation, C is searched from 100
to 103 , γ is searched from 10−4 to 100 .
For all the following experiments, each experiment
was performed 50 times, and each sample was divided
randomly according to the ratio of 80% train set and
20% test set, the final result is the average of the 50
experiments.

5.2

Comparison among Tracking data
features

We reconstruct the calculations of features like A,
D, E based on equation 1, equation 2 and equation

Figure 8: A+D+T confusion matrix
3. We add the calculation for feature T based on
equation 4. Then we use the dataset from Marin et
al. and our dataset to validate the Fingertips Tip
Distance(T) feature, the results as shown in table 1.
From table 1, we observe that both A+D+T and
A+D+E+T outperforms A+D+E more than 1% on
both of datasets, whereas the difference between
A+D+T and A+D+E+T is about 0.2%, the difference is within normal range considering the different distribution of dataset. So it can be concluded
that the feature combination of A+D+T is better
than A+D+E, the confusion matrix from A+D+T
is shown in Fig 8.

5.3

Feature fusion

We first try simply concatenate feature A+D+T
from tracking data and HOG feature from sensor images(we call 1x HOG), and compare performance on
our dataset, the results as shown in table 2.
From table 2, A+D+T+1x HOG outperforms sin-

Accuracy

A+D+T
81.54%

HOG
96.15%

A+D+T+1x HOG
98.08%

Table 2: Feature fusion

Page 4

Ours
59.50%
83.37%
82.30%
83.60%
83.36%

on our dataset of 99.42%, the confusion matrix as
shown in Fig 10.

6

Figure 9: Different weight coefficient for HOG feature

Figure 10: Confusion matrix for A+D+T+5x
HOG+80% PCA
gle HOG about 2%, we further explore the performance of different A+D+T+Kx HOG, K = 1, · · · , 9,
the results as shown in Fig 9.
The experimental results indicate that the model
achieves the best overall performance with 4x HOG
and 5x HOG and retains the accuracy of about 98%
even when dimension is reduced to 60% by PCA.
When K < 4, as dimension decreases, the overall
accuracy of the model shows a downward trend, especially 1x HOG. When K > 5, as dimension decreases,
the overall accuracy of the model shows an upward
trend, as can been seen from 9x HOG. We conclude
that when K = 4orK = 5, the variance of the features can be preserved even for significant dimension
reduction by PCA, in our experiment, we only explore
K = 1, · · · , 9, how to get a better K still remains as
a problem.
Finally, we perform PCA preserving 80% variance
on A+D+T+5x HOG, and we get the best accuracy

CONCLUSIONS AND FUTURE WORKS

In this paper, we deeply investigated the feature
extraction of Leap Motion Controller tracking data,
and propose a new feature called Fingertips Tip Distance which introduce 1% more accuracy on two
datasets. Meanwhile, we got higher accuracy by combining HOG feature extracted from binarized and
undistorted sensor images and tracking data features.
We also discussed the feasibility of linearly magnified
HOG feature by different weight coefficients. Finally,
we used PCA to perform dimension reduction on features with different coefficients.
In future work, we will further explore the characteristics of tracking data, we think the characteristics
of the remaining joints will also affect the accuracy
of the overall classification due to the correlation between joints. we perform feature fusion with different
weight coefficients in this paper, the results are considerable, but how to get a better weight coefficient
K still remains as a problem which is also what we
will do in the future. At the same time, we will study
the interaction between the system and virtual reality
application scenarios.

References
[1] Siddharth S Rautaray and Anupam Agrawal. Vision based hand gesture recognition for human
computer interaction: a survey. Artificial Intelligence Review, 43(1):1–54, 2015.
[2] Eshed Ohn-Bar and Mohan Manubhai Trivedi.
Hand gesture recognition in real time for automotive interfaces: A multimodal vision-based
approach and evaluations. IEEE transactions on
intelligent transportation systems, 15(6):2368–
2377, 2014.
[3] Chengde Wan, Angela Yao, and Luc Van Gool.
Hand pose estimation from local surface normals. In European Conference on Computer Vision, pages 554–569. Springer, 2016.
[4] Ankit Chaudhary, Jagdish Lal Raheja, Karen
Das, and Sonia Raheja. Intelligent approaches to
interact with machines using hand gesture recognition in natural way: a survey. arXiv preprint
arXiv:1303.2292, 2013.

Page 5

[5] William T Freeman, Ken-ichi Tanaka, Jun Ohta,
and Kazuo Kyuma. Computer vision for computer games. In Automatic Face and Gesture
Recognition, 1996., Proceedings of the Second International Conference on, pages 100–105. IEEE,
1996.
[6] Zhou Ren, Junsong Yuan, Jingjing Meng, and
Zhengyou Zhang. Robust part-based hand gesture recognition using kinect sensor. IEEE transactions on multimedia, 15(5):1110–1120, 2013.

[15] Wei Lu, Zheng Tong, and Jinghui Chu. Dynamic hand gesture recognition with leap motion controller. IEEE Signal Processing Letters,
23(9):1188–1192, 2016.
[16] Giulio Marin, Fabio Dominio, and Pietro Zanuttigh. Hand gesture recognition with leap motion
and kinect devices. In Image Processing (ICIP),
2014 IEEE International Conference on, pages
1565–1569. IEEE, 2014.

[7] Jesus Suarez and Robin R Murphy. Hand gesture
recognition with depth images: A review. In Roman, 2012 IEEE, pages 411–417. IEEE, 2012.
[8] Poonam Suryanarayan, Anbumani Subramanian, and Dinesh Mandalapu. Dynamic hand
pose recognition using depth data. In Pattern Recognition (ICPR), 2010 20th International Conference on, pages 3105–3108. IEEE,
2010.
[9] Fabio Dominio, Mauro Donadeo, and Pietro
Zanuttigh. Combining multiple depth-based descriptors for hand gesture recognition. Pattern
Recognition Letters, 50:101–111, 2014.
[10] Yuan Yao and Yun Fu. Contour model-based
hand-gesture recognition using the kinect sensor.
IEEE Transactions on Circuits and Systems for
Video Technology, 24(11):1935–1944, 2014.
[11] Zhi-hua Chen, Jung-Tae Kim, Jianning Liang,
Jing Zhang, and Yu-Bo Yuan. Real-time hand
gesture recognition using finger segmentation.
The Scientific World Journal, 2014, 2014.
[12] Kai-ping Feng and Fang Yuan. Static hand gesture recognition based on hog characters and
support vector machines. In Instrumentation
and Measurement, Sensor Network and Automation (IMSNA), 2013 2nd International Symposium on, pages 936–938. IEEE, 2013.
[13] Frank
Weichert,
Daniel
Bachmann,
Bartholomäus Rudak, and Denis Fisseler.
Analysis of the accuracy and robustness of the
leap motion controller. Sensors, 13(5):6380–
6393, 2013.
[14] Safa Ameur, Anouar Ben Khalifa, and Mohamed Salim Bouhlel. A comprehensive leap
motion database for hand gesture recognition.
In Information and Digital Technologies (IDT),
2017 International Conference on, pages 514–
519. IEEE, 2017.

Page 6

