RGB-D-based Human Motion Recognition with Deep Learning: A Survey
Pichao Wanga,b , Wanqing Lia , Philip Ogunbonaa , Jun Wanc,∗, Sergio Escalerad
a Advanced

arXiv:1711.08362v1 [] 31 Oct 2017

Multimedia Research Lab, University of Wollongong, Australia
b Motovis Inc., Adelaide, Australia
c Center for Biometrics and Security Research (CBSR)& National Laboratory of Pattern Recognition (NLPR), Institute of Automation, Chinese
Academy of Sciences (CASIA), Beijing, China
d University of Barcelona and Computer Vision Center, Campus UAB, Barcelona, Spain

Abstract
Human motion recognition is one of the most important branches of human-centered research activities. In recent
years, motion recognition based on RGB-D data has attracted much attention. Along with the development in artificial
intelligence, deep learning techniques have gained remarkable success in computer vision. In particular, convolutional
neural networks (CNN) have achieved great success for image-based tasks, and recurrent neural networks (RNN) are
renowned for sequence-based problems. Specifically, deep learning methods based on the CNN and RNN architectures
have been adopted for motion recognition using RGB-D data. In this paper, a detailed overview of recent advances
in RGB-D-based motion recognition is presented. The reviewed methods are broadly categorized into four groups,
depending on the modality adopted for recognition: RGB-based, depth-based, skeleton-based and multi-modal-based.
As a survey focused on the application of deep learning to RGB-D-based motion recognition, we explicitly discuss
the advantages and limitations of existing techniques. Particularly, we highlighted the methods of encoding spatialtemporal-structural information inherent in video sequence, and discuss potential directions for future research.
Keywords: Human Motion Recognition, RGB-D Data, Deep Learning, Survey

1. Introduction
Among the several human-centered research activities (e.g. human detection, tracking, pose estimation and motion
recognition) in computer vision, human motion recognition is particularly important due to its potential application in
video surveillance, human computer interfaces, ambient assisted living, human-robot interaction, intelligent driving,
etc. A human motion recognition task can be summarised as the automatic identification of human behaviours from
images or video sequences. The complexity and duration of the motion involved can be used as basis for broad
categorization into four kinds namely gesture, action, interaction and group activity. A gesture can be defined as
the basic movement or positioning of the hand, arm, body, or head that communicates an idea, emotion, etc. “Hand
waving” and “nodding” are some typical examples of gestures. Usually, a gesture has relatively short duration. An
Action is considered as a type of motion performed by a single person during a relatively short time period and uses
∗ Corresponding

author
Email addresses: pw212@uowmail.edu.au (Pichao Wang), wanqing@uow.edu.au (Wanqing Li), philipo@uow.edu.au (Philip
Ogunbona), jun.wan@nlpr.ia.ac.cn (Jun Wan), sergio@maia.ub.es (Sergio Escalera)

Preprint submitted to Computer Vision and Image Understanding

November 23, 2017

to involve multiple body parts, in contrast with the few body parts that use to be involved in gesture. An Activity is
composed by a sequence of actions. An Interaction is a type of motion performed by two actors; one actor is human
while the other may be human or an object. This implies that the interaction category will include human-human or
human-object interaction. “Hugging each other” and “playing guitar” are examples of these two kinds of interaction,
respectively. Group activity is the most complex type of activity, and it may be a combination of gestures, actions and
interactions. Necessarily, it involves more than two humans and from zero to multiple objects. Examples of group
activities would include “Two teams playing basketball” and “group meeting”.
Early research on human motion recognition was dominated by the analysis of still images or videos [1, 2, 3, 4,
5, 6]. Most of these efforts used color and texture cues in 2D images for recognition. However, the task remains
challenging due to problems posed by background clutter, partial occlusion, view-point, lighting changes, execution
rate and biometric variation. This challenge remains even with current deep learning approaches [7, 8].
With the recent development of cost-effective RGB-D sensors, such as Microsoft Kinect

TM

and Asus Xtion

TM

,

RGB-D-based motion recognition has attracted much attention. This is largely because the extra dimension, depth, is
insensitive to illumination changes and includes rich 3D structural information of the scene. Additionally, 3D positions
of body joints can be estimated from depth maps [9]. As a consequence, several methods based on RGB-D data have
been proposed and the approach has proven to be a promising direction for human motion analysis.
Several survey papers have summarized the research on human motion recognition using RGB-D data [10, 11, 12,
13, 14, 15, 16, 17]. Specifically, Chen et al. [10] focused on depth sensors, pre-processing of depth data, depth-based
action recognition methods and datasets. In their work, Ye et al. [11] presented an overview of approaches using depth
and skeleton modalities for tasks including activity recognition, head/hand pose estimation, facial feature detection and
gesture recognition. The survey presented by Aggarwal and Xia [12] summarized five categories of representations
based on 3D silhouettes, skeletal joints/body part location, local spatial-temporal features, scene flow features and local
occupancy features. The work of Cheng et al. [13] focused on RGB-D-based hand gesture recognition datasets and
summarized corresponding methods from three perspectives: static hand gesture recognition, hand trajectory gesture
recognition and continuous hand gesture recognition. In another effort Escalera et al. [15] reviewed the challenges
and methods for gesture recognition using multimodal data. Some of the surveys have focused on available datasets
for RGB-D research. For example, the work of Zhang et al. [14] described available benchmark RGB-D datasets for
action/activity recognition and included 27 single-view datasets, 10 multi-view datasets and 7 multi-person datasets.
Other works as Presti and La Cascia [16] and Han et al. [17] mainly reviewed skeleton-based representation and
approaches for action recognition. A short survey on RGB-D action recognition using deep learning was recently
presented in [8], analysing RGB and depth cues in terms of 2DCNN, 3DCNN, and Deep temporal approaches
All above surveys mainly focused on the analysis of handcrafted features. Here, we provide a comprehensive
review of RGB-D-based human motion recognition using deep learning approaches. Even while focusing on deep
learning approaches, the nature of the input data is still important. RGB-D data for human motion analysis comprises
three modalities: RGB, depth and skeleton. The main characteristic of RGB data is its shape, color and texture which
brings the benefits of extracting interesting points and optical flow. Compared to RGB videos, the depth modality is

2

RGB-D based Motion Recognition

RGB-based

Continuous/
Online

Segmented

CNNbased

RNNbased

Depth-based

Otherstructurebased

CNNbased

RNNbased

Multi-modalbased

Skeleton-based

Segmented

Continuous/
Online

Segmented

CNNbased

CNNbased

RNNbased

CNNbased

Continuous/
Online

Otherstructurebased

RNNbased

Continuous/
Online

Segmented

CNNbased

RNNbased

Otherstructurebased

RNNbased

Figure 1: Graphical representation of our categorization of the methods for RGB-D-based motion recognition using deep learning.

insensitive to illumination variations, invariant to color and texture changes, reliable for estimating body silhouette
and skeleton, and provides rich 3D structural information of the scene. Differently from RGB and depth, skeleton data
containing the positions of human joints, is a relatively high-level feature for motion recognition. It is robust to scale
and illumination changes, and can be invariant to camera view as well as human body rotation and motion speed. The
different properties of the three modalities have inspired the various methods found in the literature. For example,
optical flow-based methods with Convolutional Neural Networks (CNN) is very effective for RGB channel [18]; depth
rank pooling based-method with CNN is a good choice for depth modality [19]; sequence based method with Recurrent
Neural Networks (RNN) [20] and image-based method with CNN [21] are effective for skeleton; and scene flow-based
method using CNN are promising for RGB-D channels [22]. These methods are very effective for specific modalities,
but not always the case for all the modalities. Given these observations, this survey identified four broad categories
of methods based on the modality adopted for human motion recognition. The categories include RGB-based, depthbased, skeleton-based and multi-modal-based.
In each category, two sub-divisions are further identified, namely segmented human motion recognition and continuous/online motion recognition. For segmented motion recognition, the scenario of the problem can be simply
described as classifying a well delineated sequence of video frames as one of a set of motion types. This is in contrast
to continuous/online human motion recognition where there are no a priori given boundaries of motion execution. The
online situation is compounded by the fact that the video sequence is not recorded and the algorithm must deal with
frames as they are being captured, save for possibly a small data cache.
During the performance of a specified motion spatial information which refers to the spatial configuration of human
body at an instant of time (e.g. relative positions of the human body parts) can be identified. Similarly, there is the
temporal information which characterizes the spatial configuration of the body over time (i.e. the dynamics of the
body). Lastly, the structural information encodes the coordination and synchronization of body parts over the period
in which the action is being performed. It describes the relationship of the spatial configurations of human body across
different time slots.

3

In reviewing the various methods, consideration has been given to the manner in which the spatial, temporal and
structural information have been exploited. Hence, the survey discusses the advantages and limitations of the reviewed
methods from the spatial-temporal-structural encoding viewpoint, and suggests potential directions for future research.
A key novelty of this survey is the focus on three architectures of neural networks used in the various deep learning
methods reviewed namely CNN-based, RNN-based and other structured networks. Fig. 1 illustrates the taxonomy
underpinning this survey.
This is one of the firsts surveys dedicated to RGB-D-based human motion recognition using deep learning. Apart
from this claim, this survey distinguishes itself from other surveys through the following contributions:
• Comprehensive coverage of the most recent and advanced deep learning-based methods developed in the last
five years, thereby providing readers with a complete overview of recent research results and state-of-the-art
methods.
• Insightful categorization and analysis of methods based on the different properties of the modalities; highlight
of the pros and cons of the methods described in the reviewed papers from the viewpoint of spatial-temporalstructural encoding.
• Discussion of the challenges of RGB-D-based motion recognition; analysis of the limitations of available methods and discussion of potential research directions.
Additionally, several recently released or commonly used RGB-D-based benchmark datasets associated with deep
learning are surveyed. The main application domain of interest in this survey paper is human motion recognition
based on RGB-D data, including gesture recognition, action/activity recognition and interaction recognition. The lack
of datasets focused on RGB-D-based group activity recognition has led to paucity of research on this topic and thus
this survey does not cover this topic. Other RGB-D-based human-centered applications, such as human detection,
tracking and pose estimation, are also not the focus of this paper. For surveys on RGB-D data acquisition readers are
referred to [10, 13, 17].
Subsequent sections of the this survey are organized as follows. Commonly used RGB-D-based benchmark
datasets are described in Section 2. Sections 3 to 6 discuss methods of RGB-D-based motion recognition using deep
learning from four perspectives: RGB-based motion recognition, depth-based motion recognition, skeleton-based motion recognition and multi-modal-based motion recognition. Challenges of RGB-D-based motion recognition and
pointers to future directions are presented in Section 7. The survey provides concluding remarks in Section 8.

2. Benchmark Datasets
Over the last decade, a number of RGB-D benchmark datasets have been collected and made publicly available
for the research community. The sources of the datasets are mainly of three categories [10, 13, 17]: Motion capture
(Mocap) system, structured-light cameras (e.g. Kinect v1) and time-of-flight (ToF) cameras (e.g. Kinect v2). Hence
the modalities of the datasets cover RGB, depth, skeleton and their combinations. Recent comprehensive survey of

4

these datasets have appeared in the literature (see e.g. [13] for hand gestures and [14] for action recognition). In
the present survey only 15 large-scale datasets that have been commonly adopted for evaluating deep learning-based
methods are described. The reader is referred to Table 2 for a sample of works (publications) that have used these
datasets. For the purpose of this survey, the datasets have been divided into two groups: segmented datasets and
continuous/online datasets.
2.1. Segmented Datasets
By segmented datasets we refer to those datasets where samples correspond to a whole begin-end action/gestures,
with one segment for one action. They are mainly used for classification purposes. The following are several segmented datasets commonly used for the evaluation of methods based on deep learning.
2.1.1. CMU Mocap
CMU Graphics Lab Motion Capture Database (CMU Mocap) [23](http://mocap.cs.cmu.edu/) is one
of the earliest resources that consists of wide variety of human actions, including interaction between two subjects,
human locomotion, interaction with uneven terrain, sports, and other human actions. It is capable of recording 120 Hz
with images of 4 megapixel resolution. This dataset provides RGB and skeleton data.
2.1.2. HDM05
Motion Capture Database HDM05 [24] (http://resources.mpi-inf.mpg.de/HDM05/) was captured
by an optical marker-based technology with the frequency of 120 Hz, which contains 2337 sequences for 130 actions
performed by 5 non-professional actors, and 31 joints in each frame. Besides skeleton data, this dataset also provides
RGB data.
2.1.3. MSR-Action3D
MSR-Action3D [25] (http://www.uow.edu.au/˜wanqing/#MSRAction3DDatasets) is the first public benchmark RGB-D action dataset collected using Kinect TM sensor by Microsoft Research, Redmond and University
of Wollongong in 2010. Ten subjects performed these actions three times. All the videos were recorded from a fixed
point of view and the subjects were facing the camera while performing the actions. The background of the dataset
was removed by some post-processing. Specifically, if an action needs to be performed with one arm or one leg, the
actors were required to perform it using right arm or leg.
2.1.4. MSRC-12
MSRC-12 dataset [26](http://research.microsoft.com/en-us/um/cambridge/projects/msrc12/)
was collected by Microsoft Research Cambridge and University of Cambridge in 2012. The authors provided three
familiar and easy to prepare instruction modalities and their combinations to the participants. The modalities are (1)
descriptive text breaking down the performance kinematics, (2) an ordered series of static images of a person performing the gesture with arrows annotating as appropriate, and (3) video (dynamic images) of a person performing the
gesture. There are 30 participants in total and for each gesture, the data were collected as: Text (10 people), Images
5

(10 people), Video (10 people), Video with text (10 people), Images with text (10 people). The dataset was captured
using one Kinect TM sensor and only the skeleton data are made available.
2.1.5. MSRDailyActivity3D
MSRDailyActivity3D Dataset [27](http://www.uow.edu.au/˜wanqing/#MSRAction3DDatasets)
was collected by Microsoft and the Northwestern University in 2012 and focused on daily activities. The motivation
was to cover human daily activities in the living room. The actions were performed by 10 actors while sitting on the
sofa or standing close to the sofa. The camera was fixed in front of the sofa. In addition to depth data, skeleton data
are also recorded, but the joint positions extracted by the tracker are very noisy due to the actors being either sitting
on or standing close to the sofa.
2.1.6. UTKinect
UTKinect dataset [28](http://cvrc.ece.utexas.edu/KinectDatasets/HOJ3D.html) was collected
by the University of Texas at Austin in 2012. Ten types of human actions were performed twice by 10 subjects. The
subjects performed the actions from a variety of views. One challenge of the dataset is due to the actions being performed with high actor-dependent variability. Furthermore, human-object occlusions and body parts being out of the
field of view have further increased the difficulty of the dataset. Ground truth in terms of action labels and segmentation
of sequences are provided.
2.1.7. G3D
Gaming 3D dataset (G3D) [29](http://dipersec.king.ac.uk/G3D/) captured by Kingston University
in 2012 focuses on real-time action recognition in gaming scenario. It contains 10 subjects performing 20 gaming
actions. Each subject performed these actions thrice. Two kinds of labels were provided as ground truth: the onset and
offset of each action and the peak frame of each action.
2.1.8. SBU Kinect Interaction Dataset
SBU Kinect Interaction Dataset [30](http://www3.cs.stonybrook.edu/˜kyun/research/kinect_
interaction/index.html ) was collected by Stony Brook University in 2012. It contains eight types of interactions. All videos were recorded with the same indoor background. Seven participants were involved in performing
the activities which have interactions between two actors. The dataset is segmented into 21 sets and each set contains
one or two sequences of each action category. Two kinds of ground truth information are provided: action labels of
each segmented video and identification of “active” actor and “inactive” actor.
2.1.9. Berkeley MHAD
Berkeley Multimodal Human Action Database (Berkeley MHAD) [31](http://tele-immersion.citris-uc.
org/berkeley_mhad#dl), collected by University of California at Berkeley and Johns Hopkins University in
2013, was captured in five different modalities to expand the fields of application. The modalities are derived from:

6

optical mocap system, four multi-view stereo vision cameras, two Microsoft Kinect v1 cameras, six wireless accelerometers and four microphones. Twelve subjects performed 11 actions, five times each. Three categories of
actions are included: (1) actions with movement in full body parts, e.g., jumping in place, jumping jacks, throwing,
etc., (2) actions with high dynamics in upper extremities, e.g., waving hands, clapping hands, etc. and (3) actions
with high dynamics in lower extremities, e.g., sit down, stand up. The actions were executed with style and speed
variations.
2.1.10. Northwestern-UCLA Multiview Action 3D
Northwestern-UCLA Multiview Action 3D [32](http://users.eecs.northwestern.edu/˜jwa368/
my_data.html) was collected by Northwestern University and University of California at Los Angles in 2014. This
dataset contains data taken from a variety of viewpoints. The actions were performed by 10 actors and captured by
three simultaneous KinectTM v1 cameras.
2.1.11. ChaLearn LAP IsoGD
ChaLearn LAP IsoGD Dataset [33] (http://www.cbsr.ia.ac.cn/users/jwan/database/isogd.
html) is a large RGB-D dataset for segmented gesture recognition, and it was collected by Kinect v1 camera. It
includes 47933 RGB-D depth sequences, each RGB-D video representing one gesture instance. There are 249 gestures
performed by 21 different individuals. The dataset is divided into training, validation and test sets. All three sets consist
of samples of different subjects to ensure that the gestures of one subject in the validation and test sets will not appear
in the training set.
2.1.12. NTU RGB+D
NTU RGB+D Dataset [34](https://github.com/shahroudy/NTURGB-D) is currently the largest action
recognition dataset in terms of the number of samples per action. The RGB-D data is captured by Kinect v2 cameras.
The dataset has more than 56 thousand sequences and 4 million frames, containing 60 actions performed by 40 subjects
aging between 10 and 35. It consists of front view, two side views and left, right 45 degree views.
2.2. Continuous/Online Datasets
Continuous/online datasets refer to those datasets where each video sequence may contain one or more actions/gestures,
and the segmented position between different motion classes are unknown. These datasets are mainly used for action
detection, localization and online prediction. There are few datasets for this type.
2.2.1. ChaLearn2014 Multimodal Gesture Recognition

ChaLearn2014 Multimodal Gesture Recognition [35] (http://gesture.chalearn.org/2014-looking-at-people-c
is multi-modal dataset collected by Kinect v1 sensor, including RGB, depth, skeleton and audio modalities. In all sequences, a single user is recorded in front of the camera, performing natural communicative Italian gestures. The
starting and ending frames for each gesture are annotated along with the gesture class label. It contains nearly 14K

7

manually labeled (beginning and ending frame) gesture performances in continuous video sequences, with a vocabulary of 20 Italian gesture categories. There are 1, 720, 800 labeled frames across 13, 858 video fragments of about 1
to 2 minutes sampled at 20 Hz. The gestures are performed by 27 different individuals under diverse conditions; these
include varying clothes, positions, backgrounds and lighting.
2.2.2. ChaLearn LAP ConGD
The ChaLearn LAP ConGD Dataset [33] (http://www.cbsr.ia.ac.cn/users/jwan/database/congd.
html) is a large RGB-D dataset for continuous gesture recognition. It was collected by Kinect v1 sensor and includes
47933 RGB-D gesture instances in 22535 RGB-D gesture videos. Each RGB-D video may contain one or more gestures. There are 249 gestures performed by 21 different individuals. The dataset is divided into training, validation
and test sets. All three sets consist of samples of different subjects to ensure that the gestures of one subject in the
validation and test sets will not appear in the training set.
2.2.3. PKU-MMD
PKU-MMD [36] (http://www.icst.pku.edu.cn/struct/Projects/PKUMMD.html) is a large scale
dataset for continuous multi-modality 3D human action understanding and covers a wide range of complex human activities with well annotated information. It was captured via the Kinect v2 sensor. PKU-MMD contains 1076 long
video sequences in 51 action categories, performed by 66 subjects in three camera views. It contains almost 20,000 action instances and 5.4 million frames in total. It provides multi-modality data sources, including RGB, depth, Infrared
Radiation and Skeleton.
Table 2 shows the statistics of publicly available benchmark datasets that are commonly used for evaluation of
deep learning-based algorithms. It can been seen that the surveyed datasets cover a wide range of different types of
actions including gestures, simple actions, daily activities, human-object interactions, human-human interactions. It
also covers both segmented and continuous/online datasets, with different acquisition devices, modalities, and views.
Sample images from different datasets are shown in Fig. 2.
In this section, we introduce the deep learning concepts and architectures that are relevant or have been applied to
RGB-D-based motion recognition. Readers who are interested in more background and techniques are referred to the
book by [41].

3. RGB-based Motion Recognition with Deep Learning
RGB is one important channel of RGB-D data with characteristics including shape, color and texture that bear rich
features. These properties also make it effective to directly use texture-driven feature extraction networks, such as 2D
1 Also

contains Audio and Accelerometer signals.
contains IR videos.
3 Also contains Audio signal.
4 Also contains IR videos.
2 Also

8

Table 1: Statistics of public available benchmark datasets that are commonly used for evaluation with deep learning. Notation for the header: Seg:
Segmented, Con:Continuous, D: Depth, S:Skeleton, #:number of, JI:Jaccard Index.

Dataset

year

Acquisition

Seg/Con

Modality

#Class

#Subjects

#Samples

#Views

Metric

device
CMU Mocap

2001

Mocap

Seg

RGB,S

45

144

2,235

1

Accuracy

HDM05

2007

Mocap

Seg

RGB,S

130

5

2337

1

Accuracy

MSR-Action3D

2010

Kinect v1

Seg

S,D

20

10

567

1

Accuracy

MSRC-12

2012

Kinect v1

Seg

S

12

30

594

1

Accuracy

2012

Kinect v1

Seg

RGB,D,S

16

10

320

1

Accuracy

UTKinect

2012

Kinect v1

Seg

RGB,D,S

10

10

200

1

Accuracy

G3D

2012

Kinect v1

Seg

RGB,D,S

5

5

200

1

Accuracy

2012

Kinect v1

Seg

RGB,D,S

7

8

300

1

Accuracy

Seg

RGB,D,S1

12

12

660

4

Accuracy

MSR
DailyActivity3D

SBU Kinect
Interaction
Berkeley MHAD

2013

Mocap
Kinect v1

Multiview

2014

Kinect v1

Seg

RGB,D,S

10

10

1475

3

Accuracy

2016

Kinect v1

Seg

RGB,D

249

21

47933

1

Accuracy

NTU RGB+D

2016

Kinect v2

Seg

RGB,D,S2

60

40

56880

80

Accuracy

ChaLearn2014

2014

Kinect v1

Con

RGB,D,S3

20

27

13858

1

Action3D
ChaLearn LAP
IsoGD

Accuracy
JI etc.

ChaLearn LAP

2016

Kinect v1

Con

RGB,D

249

21

22535

1

JI

2017

Kinect v2

Con

RGB,D,S4

51

66

1076

3

JI etc.

ConGD
PKU-MMD

9

CMU Mocap

HDM05

MSR-Action3D

MSRC-12

MSR
DailyActivity3D

UTKinect

G3D

SBU Kinect
Interaction

Berkeley
MHAD

Multiview
Action3D

ChaLearn Lap
IsoGD

NTU
RGB+D

ChaLearn2014

ChaLearn Lap
ConGD

PKU-MMD

10
Figure 2: Sample images from different datasets.

Figure 3: C3D net has 8 convolutions, 5 max-poolings, and 2 fully connected layers, followed by a softmax output layer. All 3D convolution
kernels are 3×3×3 with stride 1 in both spatial and temporal dimensions. Number of filters are indicated in each box. The 3D pooling layers are
as indicated from pool1 to pool5. All pooling kernels are 2×2×2, except for pool1 which is 1×2×2. Each fully connected layer has 4096 output
units. Figure from [47].

Figure 4: Different approaches for fusing information over temporal dimension through the network. Red, green and blue boxes indicate convolutional, normalization and pooling layers respectively. In the Slow Fusion model, the depicted columns share parameters. Figure from [45].

CNN [42, 43, 44] to extract frame-level spatial information. Even though most of the surveyed methods for this section are not adapted to RGB-D-based datasets, we argue that the following methods could be directly adapted to RGB
modality of RGB-D datasets. We define three categories namely, CNN-based, RNN-based and other-architecturebased approaches for segmented motion recognition; the first two categories are for continuous/online motion recognition.
3.1. Segmented Motion Recognition
3.1.1. CNN-based Approach
For this group of methods, currently there are mainly four approaches to encode spatial-temporal-structural information. The first approach applies CNN to extract features from individual frames and later, fuse the temporal
information. For example [45] investigated four temporal fusion methods, and proposed the concept of slow fusion
where higher layers get access to progressively more global information in both spatial and temporal dimensions. The
implementation extends the connectivity of all convolutional layers in time and carry out temporal convolutions (see
Fig. 4). Furthermore, several temporal pooling methods have been explored and the suggestion is that max pooling in
the temporal domain is preferable [46].
The second approach extends convolutional operation into temporal domain. In one such implementation, Ji
et al. [48] proposed 3D-convolutional networks using 3D kernels (filters extended along the time axis) to extract
features from both spatial and temporal dimensions. This work empirically showed that the 3D-convolutional networks
outperform their 2D frame-based counterparts. With modern deep architectures, such as VGG [43], and large-scale
supervised training datasets, such as Sports-1M [45], Tran et al [47] extended the work presented in [48] by including
3D pooling layers, and proposed a generic descriptor named C3D by averaging the outputs of the first fully connected

11

Figure 5: Two-stream architecture for RGB-based motion recognition. Figure from [56].

layer of the networks (see Fig. 3). However, both of these works break the video sequence into short clips and
aggregate video-level information by late score fusion. This is likely to be suboptimal when considering some long
action sequence, such as walking or swimming that lasts several seconds and spans tens or hundreds of video frames.
To handle this problem, Varol et al. [49] investigated the learning of long-term video representations and proposed
the Long-term Temporal Convolutions (LTC) at the expense of decreasing spatial resolution to keep the complexity
of networks tractable. Despite the fact that this is straightforward and mainstream, extending spatial kernels to 3D
spatio-temporal derivative inevitably increases the number of parameters of the network. To relieve the drawbacks of
3D kernels, Sun et al. [50] factorized a 3D filter into a combination of 2D and 1D filters.
The third approach is to encode the video into dynamic images that contain the spatio-temporal information and
then apply CNN for image-based recognition. Bilen et al. [51] proposed to adopt rank pooling [52] to encode the
video into one dynamic set of images and used pre-trained models over ImageNet [42] for fine-tuning (see Figure. 6).
The end-to-end learning methods with rank pooling has also been proposed in [51, 53]. Hierarchical rank pooling [54]
is proposed to learn higher order and non-linear representations compared to the original work. Generalized rank
pooling [55] is introduced to improve the original method via a quadratic ranking function which jointly provides a
low-rank approximation to the input data and preserves their temporal order in a subspace.
Besides the above works that aim to adopt one network to exploit both spatio-temporal information contained in the
video, the fourth approach separates the two factors and adopt multiple stream networks. Simonyan and Zisserman [56]
proposed one spatial stream network fed with raw video frames, and one temporal stream network accepting optical
flow fields as input. The two streams are fused together using the softmax scores (see Figure. 5 for the two-stream
architecture). Wang et al. [57] extended the two-stream networks by integrating improved trajectories [58], where
trajectory-constrained sampling and pooling are used to encode deep features learned from deep CNN architecture,
into effective descriptors. To incorporate long-range temporal structure using the two-stream networks, Wang et
al. [59] devised a temporal segment network (TSN) that uses a sparse sampling scheme to extract short snippets
over a long video sequence. With the removal of redundancy from consecutive frames and a segmental structure,
aggregated information is obtained from the sampled snippets. To reduce the expensive calculation of optical flow,
Zhang et al. [60] accelerated this two stream structure by replacing optical flow with motion vector which can be
obtained directly from compressed videos without extra calculation. Wang et al. [61] leveraged semantic cues in

12

Figure 6: Rank pooling encodes the RGB video into one dynamic image and CNN is adopted for feature extraction and classification. Figure
from [51].

video by using a two-stream semantic region-based CNNs (SR-CNNs) to incorporate human/object detection results
into the framework. In their work, Chéron et al. [62] exploit spatial structure of the human pose and extract a posebased convolutional neural network (P-CNN) feature from both RGB frames and optical flow for fine-grained action
recognition. The work presented in [63] formulated the problem of action recognition from a new perspective and
model an action as a transformation which changes the state of the environment before the action to the state after
the action. They designed a Siamese network which models the action as a transformation on a high-level feature
space based on the two-stream model. Based on the two-stream framework, [64] proposed a key volume mining deep
framework for action recognition, where they identified key volumes and conducted classification simultaneously.
Inspired by the success of Residual Networks (ResNets) [44], Feichtenhofer et al. [65] injected residual connections
between the two streams to allow spatial-temporal interaction between them. Instead of using optical flow for temporal
stream, [66] adopted Motion History Image (MHI) [67] as the motion clue. The MHI was combined with RGB
frames in a spatio-temporal CNN for fined grained action recognition. However, all the methods reviewed above
incorporated the two streams from separate training regimes; any registration of the two streams was neglected. In
order to address this gap and propose a new architecture for spatial-temporal fusion of the two streams Feichtenhofer
et al. [68] investigated three aspects of fusion for the two streams: (i) how to fuse the two networks with consideration
for spatial registration, (ii) where to fuse the two networks and, (iii) how to fuse the networks temporally. One of their
conclusions was that the results suggest the importance of learning correspondences between highly abstract ConvNet
features both spatially and temporally.
3.1.2. RNN-based Approach
For RNN-based approach, Baccouche et al. [69] tackled the problem of action recognition through a cascade of 3D
CNN and LSTM, in which the two networks were trained separately. Differently from the separate training, Donahue
et al. [70] proposed one Long-term Recurrent Convolutional Network (LRCN) to exploit end-to-end training of the

13

two networks( see illustration in Figure. 7). To take full advantage of both CNN and RNN, Ng et al. [46] aggregated
CNN features with both temporal pooling and LSTM for temporal exploitation, and fused the output scores from the
feature pooling and LSTM network to conduct final action recognition. Pigou et al. [71] proposed an end-to-end trainable neural network architecture incorporating temporal convolutions and bidirectional LSTM for gesture recognition.
This provided opportunity to mine temporal information that is much discriminative for gesture recognition. In their
work, Sharma et al. [72] proposed a soft attention model for action recognition based on LSTM (see Figure. 8). The
attention model learns the parts in the frames that are relevant for the task at hand and attaches higher importance to
them. Previous attention-based methods have only utilized video-level category as supervision to train LSTM. This
strategy may lack a detailed and dynamical guidance and consequently restrict their capacity for modelling complex
motions in videos. Du et al. [73] address this problem by proposing a recurrent pose-attention network (RPAN) for
action recognition in videos, which can adaptively learn a highly discriminative pose-related feature for every-step
action prediction of LSTM. To take advantage of both Fisher Vector [74] and RNN, Lev et al. [75] introduced a
Recurrent Neural Network Fisher Vector (RNN-FV) where the GMM probabilistic model in the fisher vector is replaced by a RNN and thus avoids the need for the assumptions of data distribution in the GMM. Even though RNN
is remarkably capable of modeling temporal dependences, it lacks an intuitive high-level spatial-temporal structure.
The spatio-temporal-structural information has been mined by Jain et al. [76] through a combination of the powers of
spatio-temporal graphs and RNN for action recognition. Recently, Sun et al. [77] proposed a Lattice-LSTM (L2STM)
network, which extends LSTM by learning independent hidden state transitions of memory cells for individual spatial
locations. This method effectively enhances the ability to model dynamics across time and addresses the non-stationary
issue of long-term motion dynamics without significantly increasing the model complexity. Differently from previous
methods that using only feedforward connections, Shi et al. [78] proposed a biologically-inspired deep network, called
ShuttleNet1. Unlike traditional RNNs, all processors inside ShuttleNet are connected in a loop to mimic the human
brains feedforward and feedback connections. In this manner, the processors are shared across multiple pathways in
the loop connection. Attention mechanism is then employed to select the best information flow pathway.
3.1.3. Other-architecture-based Approach
Besides the commonly used CNN- and RNN-based methods for motion recognition from RGB modality, there
are several other structures that have been adopted for this task. Jhuang et al. [81] used a feedforward hierarchical
template matching architecture for action recognition with pre-defined spatio-temporal filters in the first layer. In his
thesis, Chen [82] adopted the convolutional RBM (CRBM) as the basic processing unit and proposed the so-called
space-time Deep Belief Network (ST-DBN) that alternates the aggregation of spatial and temporal information so
that higher layers capture longer range statistical dependencies in both space and time. Taylor et al. [83] extended
the Gated RBM (GRBM) [84] to convolutional GRBM (convGRBM) that shares weights at all locations in an image
and inference is performed through convolution. Le et al. [85] presented an extension of the independent subspace
analysis algorithm [86] to learn invariant spatio-temporal features from unlabeled video data. They scale up the
original ISA to larger input data by employing two important ideas from convolutional neural networks: convolution
and stacking. This convolutional stacking idea enables the algorithm to learn a hierarchical representation of the data
14

Figure 7: LRCN processes the variable-length visual input with a CNN, whose outputs are fed into a stack of recurrent sequence models. The output
is a variable-length prediction. Figure from [70].

suitable for recognition. Yan et al. [87] proposed Dynencoder, a three layer auto-encoder, to capture video dynamics.
Dynencoder is shown to be successful in synthesizing dynamic textures, and one can think of a Dynencoder as a
compact way of representing the spatio-temporal information of a video. Similarly, Srivastava et al. [80] introduced a
LSTM autoencoder model. The LSTM autoencoder model consists of two RNNs, namely, the encoder LSTM and the
decoder LSTM. The encoder LSTM accepts a sequence as input and learns the corresponding compact representation.
The states of the encoder LSTM contain the appearance and dynamics of the sequence. The decoder LSTM receives the
learned representation to reconstruct the input sequence. Inspired by the Generative Adversarial Networks (GAN) [88],
Mathieu et al. [89] adopted the adversarial mechanism to train a multi-scale convolutional network to generate future
frames given an input sequence. To deal with the inherently blurry predictions obtained from the standard Mean
Squared Error (MSE) loss function, they proposed three different and complementary feature learning strategies: a
multi-scale architecture, an adversarial training method, and an image gradient difference loss function.
3.2. Continuous/Online Motion Recognition
Most of the action recognition methods reviewed above rely heavily on segmented videos for model training.
However, it is very expensive and time-consuming to acquire a large-scale trimmed video dataset. The availability
of untrimmed video datasets (e.g. [37, 39, 38, 33, 36]) have encouraged research and challenges/contests in motion
recognition in this domain.

15

Figure 8: The CNN takes the video frames as its input and produces a feature tube. The model computes the current input xt as an average of the
feature slices weighted according to the location softmax I t . At each time-step t, the recurrent network takes a feature slice xt as input. It then
propagates xt through three layers of LSTMs and predicts the next location probabilities I t+1 and the class label y t . Figure from [72].

3.2.1. CNN-based Approach
Inspired by the success of region-proposal-based object detection using R-CNN [90, 91, 92], several proposalbased action recognition methods from untrimmed video are proposed. These methods first generate a reduced number
of candidate temporal windows, and then an action classifier discriminates each proposal independently into one of the
actions of interest. For instance, based on the two-stream concept [56], [93] classified frame-based region proposals
of interest using static and motion cues. The regions are then linked across frames based on the predictions and
their spatial overlap; thus producing action tubes respectively for each action and video. Weinzaepfel et al. [94]
also started from the frame-level proposals, selected the highest scoring ones, tracked them throughout the video,
and adopted a multi-scale sliding window approach over tracks to detect the temporal content of an action. Shou et
al. [95] proposed a multi-stage segment-based 3D CNNs to generate candidate segments, that are used to recognize
actions and localize temporal boundaries. Peng and Schimd [96] generated rich proposals from both RGB and optical
flow data by using region proposal networks for frame-level action detection, and stacked optical flows to enhance
the discriminative power of motion R-CNN. Wang et al. [97] proposed an UntrimmedNet to generate clip proposals
that may contain action instances for untrimmed action recognition. Based on these clip-level representations, the
classification module aims to predict the scores for each clip proposal and the selection module tries to select or rank
those clip proposals. Similarly in the same direction, Zhao et al. [98] adopted explicit structural modeling in the
temporal dimension. In their model, each complete activity instance is considered as a composition of three major
stages, namely starting, course, and ending, and they introduced structured temporal pyramid pooling to produce a
global representation of the entire proposal. Differently from previous methods, Zhu et al. [99] proposed a framework
that integrates the complementary spatial and temporal information into an end-to-end trainable system for video
action proposal, and a novel and efficient path trimming method is proposed to handle untrimmed video by examining
actionness and background score pattern without using extra detectors. To generalize R-CNN from 2D to 3D, Hou et

16

FPS
134

2
可

•

•

Figure 9: The Deep Action Proposals algorithm can localize segments of varied duration around actions occurring along a video without exhaustively
exploring multiple temporal scales. Figure from [79].

Figure 10: The LSTM autoencoder model and LSTM future predictor model. Figure from [80].

al. [100] proposed an end-to-end 3D CNN-based approach for action detection in videos. A Tube Proposal Network
was introduced to leverage skip pooling in temporal domain to preserve temporal information for action localization
in 3D volumes, and Tube-of-Interest pooling layer was proposed to effectively alleviate the problem with variable
spatial and temporal sizes of tube proposals. Saha et al. [101] proposed a deep net framework capable of regressing
and classifying 3D region proposals spanning two successive video frames. The core of the framework is an evolution
of classical region proposal networks (RPNs) to 3D RPNs. Similarly, Kalogeiton et al. [102] extended the Single Shot
MultiBox Detector (SSD) [103] framework from 2D to 3D by proposing an Action Tubelet detector. In order to quickly
and accurately generate temporal action proposals, Gao et al. [104] proposed a Temporal Unit Regression Network
(TURN) model, that jointly predicts action proposals and refines the temporal boundaries by temporal coordinate
regression using CNN. Similarly, Singh et al. [105] designed an efficient online algorithm to incrementally construct
and label ’action tubes’ from the SSD frame level detections, making it the first real-time (up to 40fps) system able
to perform online S/T action localisation on the untrimmed videos. Besides the proposal-based methods discussed
17

飞

above, Lea et al. [106] introduced a new class of temporal models, called Temporal Convolutional Networks (TCNs),
that use a hierarchy of temporal convolutions to perform fine-grained action segmentation or detection.

Figure 11: The problem of weakly supervised action labelling is tackled where only the order of the occurring actions in given during training. The
temporal model is trained by maximizing the probability of all possible frame-to-label alignments. At testing time, no annotation is given. The
learned model encodes the temporal structure of videos and could predict the actions without further information. Figure from [107].

3.2.2. RNN-based Approach
Apart from the proposal-based methods that use CNN, there are several proposal-based methods using RNN for
temporal modeling. Escorcia et al. [79] introduced the Deep Action Proposals (DAPs) that generate temporal action
proposals from long untrimmed videos for action detection and classification (see Figure. 9). They adopted C3D
network [47] for visual encoder and LSTM for sequence encoder. However, all of these methods generated proposals
by a sliding window approach, dividing the video into short overlapping temporal window, which is computationally
expensive. To reduce the number of proposals, Buch et al. [108] proposed a single-stream temporal action proposal
generation method that does not the need to divide input into short overlapping clips or temporal windows for batch
processing.
Besides the proposal-based methods, there are several methods that are proposal-free. Yeung et al. [109] proposed
an end-to-end training model that is formulated as a recurrent neural network-based agent. This agent learns a policy
for sequentially forming and refining hypotheses about action instances based on the intuition that the process of
detecting actions is naturally one of observation and refinement. They adopted two networks namely, observation
network and recurrent network, for this purpose. Singh et al. [110] presented a multi-stream bi-directional recurrent
neural network for fine-grained action detection. They adopted a tracking algorithm to locate a bounding box around
the person and trained two streams on motion and appearance cropped to the tracked bounding box. The video
sequence was split into fixed long chunks for the input of two-stream networks, and bi-directional LSTM was used
to model long-term temporal dynamics within and between actions. Ma et al. [111] introduced a novel ranking loss
within the RNN objective so that the trained model better captures progression of activities. The ranking loss constrains
the detection score of the correct category to be monotonically non-decreasing as the activity progresses. The same
time, the detection score margin between the correct activity category and all other categories is monotonically non18

decreasing. Huang et al. [107] proposed a weakly-supervised framework for action labeling in video ( see Figure 11),
where only the order of occurring actions is required during training. They proposed an Extended Connectionist
Temporal Classification (ECTC) framework to efficiently evaluate all possible alignments between the input and label
sequences via dynamic programming and explicitly enforce their consistency with frame-to-frame visual similarities.
Taking inspiration from classical linear dynamic systems theory for modeling time series, Dave et al. [112] derived
a series of recurrent neural networks that sequentially make top-down predictions about the future and then correct
those predictions with bottom-up observations. Their predictive-corrective architecture allows the incorporation of
insights from time-series analysis: adaptively focus computation on “surprising” frames where predictions require
large corrections; simplify learning in that only “residual-like” corrective terms need to be learned over time and
naturally decorrelate an input stream in a hierarchical fashion, producing a more reliable signal for learning at each
layer of a network.

4. Depth-based Motion Recognition with Deep Learning
Compared with RGB videos, the depth modality is insensitive to illumination variations, invariant to color and
texture changes, reliable for estimating body silhouette and skeleton, and provides rich 3D structural information
of the scene. However, there are only few published results on depth based action recognition using deep learning
methods. Two reasons can be adduced for this situation. First, the absence of color and texture in depth maps weakens
the discriminative representation power of CNN models which are texture-driven feature extractor and classifier [113].
Second, existing depth data is relatively small-scale. The conventional pipelines are purely data-driven and learn
representation directly from the pixels. Such model is likely to be at risk of overfitting when the network is optimized
on limited training data. Currently, there are only CNN-based methods for depth-based motion recognition.
4.1. Segmented Motion Recognition
4.1.1. CNN-based Approach
Wang et al. [114, 115] took advantage of the representation power of CNN on texture images and at the same
time enlarge available training data by encoding depth map sequences into texture color images using the concepts
of Depth Motion Maps (DMM) [116] and pseudo-coloring; training data was enlarged by scene rotation on the 3D
point cloud (see Fig. 12). Inspired by the promising results achieved by rank pooling method [51] on RGB data, Wang
et al. [19] encoded the depth map sequences into three kinds of dynamic images with rank pooling: Dynamic Depth
Images (DDI), Dynamic Depth Normal Images (DDNI) and Dynamic Depth Motion Normal Images (DDMNI). These
three representations capture the posture and motion information from three different levels for gesture recognition.
Differently from the above texture image encoding method, Rahmani and Mian [117] proposed a cross-view action
recognition based on depth sequence. Their method comprises two steps: (i) learning a general view-invariant human
pose model from synthetic depth images and, (ii) modeling the temporal action variations. To enlarge the training
data for CNN, they generated the training data synthetically by fitting realistic synthetic 3D human models to real
mocap data and then rendering each pose from a large number of viewpoints. For spatio-temporal representation, they
19

Convolutional Neural Networks

11
11

55

rotation
and
n
temporal
scales

27

13
3

5

256

13
3

13

27

55

13
3

3
3

3

5

c

13

13
384

384

256

4096 4096

96

WHDMMf
Convolutional Neural Networks

11
11

rotation
and
n
temporal
scales

55

27

256

13

3
3

13

27

class
score
fusion

13

3
3

3

5
55

c

13

13
384

384

256

4096

4096

256

4096

4096

96

WHDMMs

Input
depth
maps

13

3

5

Convolutional Neural Networks

1

111
1
11
1

55

rotation
and
n
temporal
scales

27

13

3

5

256

13

3
3

13

27

55

13

3
3

3

5

384

c

13

13
384

96

WHDMMt
conv1

conv2

conv3

conv4

conv5

fc6

fc7 fc8

fusion

Figure 12: Depth map sequences are encoded into texture color images by using the concepts of Depth Motion Maps (DMM) [116] and pseudocoloring, and at the same time enlarged the training data by scene rotation on the 3D point cloud. Three channel of CNN are adopted for feature
extraction and classification. Figure from [115].

used group sparse Fourier Temporal Pyramid which encodes the action-specific discriminative output features of the
proposed human pose model.
4.2. Continuous/Online Motion Recognition
4.2.1. CNN-based Approach
For continuous gesture recognition, Wang et al. [118] first segmented the continuous depth sequence into segmented sequences using quantity of movement (QOM) [119], and then adopted improved DMM (IDMM) to encode
the dynamics of depth sequences into texture images for large-scale continuous gesture recognition.

5. Skeleton-based Motion Recognition with Deep Learning
Differently from RGB and depth, skeleton data contains the positions of human joints, which can be considered
relatively high-level features for motion recognition. Skeleton data is robust to scale and illumination changes, and
can be invariant to camera view as well as human body rotation and motion speed. Currently, there are mainly
three approaches to skeleton-based motion recognition using deep learning: (i) RNN-based, (ii) CNN-based and
other-architecture-based approaches for segmented motion recognition and, (iii) RNN-based approaches for continuous/online motion recognition.

20

6FRUHV

&RQY1HWV
)URQW
&KDQQHO)URQW

7RS
6NHOHWRQ
6HTXHQFHV

6FRUHV

&RQY1HWV

)XVLRQ

$FFXUDF\

&KDQQHO7RS

6LGH

6FRUHV

&RQY1HWV
&KDQQHO6LGH

Figure 13: The JTM framework for skeleton-based motion recognition with CNN. Figure from [21].

5.1. Segmented Motion Recognition
5.1.1. CNN-based Approach
The main step in this approach is to convert the skeleton sequences into images where the spatio-temporal information is reflected in the image properties including color and texture. Duet al. [120] represented a skeleton sequence
as a matrix by concatenating the joint coordinates at each instant and arranging the vector representations in a chronological order. The matrix is then quantified into an image and normalized to handle the variable-length problem. The
final image is fed into a CNN model for feature extraction and recognition. Wang et al. [21] proposed to encode
spatio-temporal information contained in the skeleton sequence into multiple texture images, namely, Joint Trajectory
Maps (JTM), by mapping the trajectories into HSV (hue, saturation, value) space. Pre-trained models over Imagenet
is adopted for fine-tuning over the JTMs to extract features and recognize actions. Similarly, Hou et al. [121] drew the
skeleton joints with a specific pen to three orthogonal canvases, and encodes the dynamic information in the skeleton
sequences with color encoding. Li et al. [122] proposed to encode the pair-wise distances of skeleton joints of single
or multiple subjects into texture images, namely, Joint Distance Maps (JDM), as the input of CNN for action recognition. Compared with the works reported by [21] and [121], JDM is less sensitive to view variations. liu et al. [123]
introduced an enhanced skeleton visualization method to represent a skeleton sequence as a series of visual and motion enhanced color images. They proposed a sequence-based view invariant transform to deal with the view variation
problem, and multi-stream CNN fusion method is adopted to conduct recognition. Ke et al. [124] designed vectorbased features for each body part of human skeleton sequences, which are translation, scale and rotation invariant, and
transformed the features into images to feed into CNN for learning high level and discriminative representation. In
another effort, Ke et al. [125] represented the sequence as a clip with several gray images for each channel of the 3D
coordinates, which reflects multiple spatial structural information of the joints. The images are fed to a deep CNN to
learn high-level features, and the CNN features of all the three clips at the same time-step are concatenated in a feature
vector. Each feature vector represents the temporal information of the entire skeleton sequence and one particular
spatial relationship of the joints. A Multi-Task Learning Network (MTLN) is adopted to jointly process the feature
21

,

}
) lþ>[旦旦

气~

BRì心4

习>皿出

==L

I::þI BRNN ~
I

革>匪旦旦

ω〉CJgEMGa
』

97

且飞

今I BRNN I'Ç方
Laye r1

Layer2

Layer3

Layer4

、_.._.._.._.._.._.._.._.._..-

Layer5

Layer6

Layer7

Layer8 Laye r91

Figure 14: The LSTM autoencoder model and LSTM future predictor model. Figure from [127].

vectors of all time-steps in parallel for action recognition. Kim and Reiter [126] approached the problem differently
and proposed to use the Temporal Convolutional Neural Networks (TCN) [106] for skeleton based action recognition.
They re-designed the original TCN into Res-TCN by factoring out the deeper layers into additive residual terms that
yields both interpretable hidden representations and model parameters.
5.1.2. RNN-based Approach
In this class of approaches, skeleton features are input to an RNN in order to exploit the temporal evolution. For
instance, in a series or works Du et al. [127, 128] divided the whole skeleton sequence into five parts according to
the human physical structure, and separately fed them into five bidirectional RNNs/LSTMs. As the number of layers
increases, the representations extracted by the subnets are hierarchically fused to build a higher-level representation.
The process is illustrated in Fig. 14. This method explicitly encodes the spatio-temporal-structural information into
high level representation. Veeriah et al. [129] proposed a differential gating scheme for the LSTM neural network,
which emphasizes the change in information gain caused by the salient motions between the successive frames. This
work is one of the first aimed at demonstrating the potential of learning complex time-series representations via highorder derivatives of states. Zhu et al. [130] designed two types of regularizations to learn effective features and motion
dynamics. In the fully connected layers, they introduced regularization to drive the model to learn co-occurrence
features of the joints at different layers. Furthermore, they derived a new dropout and apply it to the LSTM neurons
in the last LSTM layer, which helps the network to learn complex motion dynamics. Instead of keeping a long-term
memory of the entire body’s motion in the cell, Shahroudy et al. [34] proposed a part-aware LSTM human action
learning model (P-LSTM) wherein memory is split across part-based cells. It is argued that keeping the context of
each body part independent and representing the output of the P-LSTM unit as a combination of independent body
part context information is more efficient. Previous RNN-based 3D-action recognition methods have adopted RNN
to model the long-term contextual information in the temporal domain for motion-based dynamics representation.
However, there is also strong dependency between joints in the spatial domain. In addition the spatial configuration
of joints in video frames can be highly discriminative for 3D-action recognition task. To exploit this dependency,
Liu et al.[131] proposed a spatio-temporal LSTM (ST-LSTM) network which extends the traditional LSTM-based
22

Figure 15: The joint classification-regression RNN framework for online action detection and forecasting. Figure from [137].

learning to both temporal and spatial domains. Rather than concatenate the joint-based input features, ST-LSTM
explicitly models the dependencies between the joints and applies recurrent analysis over spatial and temporal domains
concurrently. Besides, they introduced a trust gate mechanism to make LSTM robust to noisy input data. Song
et al. [132] proposed a spatio-temporal attention model with LSTM to automatically mine the discriminative joints
and learn the respective and different attentions of each frame along the temporal axis. Similarly, Liu et al. [20]
proposed a Global Context-Aware Attention LSTM (GCA-LSTM) to selectively focus on the informative joints in the
action sequence with the assistance of global context information. Differently from previous works that adopted the
coordinates of joints as input, Zhang et al. [133] investigated a set of simple geometric features of skeleton using 3layer LSTM framework, and showed that using joint-line distances as input requires less data for training. Based on the
notion that LSTM networks with various time-step sizes can model various attributes well, Lee et al. [134] proposed an
ensemble Temporal Sliding LSTM (TS-LSTM) networks for skeleton-based action recognition. The proposed network
is composed of multiple parts containing short-term, medium- term and long-term TS-LSTM networks, respectively.
Li et al. [135] proposed an adaptive and hierarchical framework for fine-grained, large-scale skeleton-based action
recognition. This work was motivated by the need to distinguish fine-grained action classes that are intractable using
a single network, and adaptivity to new action classes by model augmentation. In the framework, multiple RNNs
are effectively incorporated in a tree-like hierarchy to mitigate the discriminative challenge and thus using a divideand-conquer strategy. To deal with large view variations in captured human actions, Zhang et al. [136] proposed
a self-regulated view adaptation scheme which re-positions the observation viewpoints dynamically, and integrated
the proposed view adaptation scheme into an end-to-end LSTM network which automatically determines the “best
observation viewpoints during recognition.
5.1.3. Other-architecture-based Approach
Besides the RNN- and CNN-based approaches, there are several other deep learning-based methods. Salakhutdinov et al. [139] proposed a new compositional learning architecture that integrates deep learning models with structured hierarchical Bayesian models. Specifically, this method learns a hierarchical Dirichlet process (HDP) [140]
23

Figure 16: Conceptual illustration of LieNet architecture. In the network structure, the data space of each RotMap/RotPooling layer corresponds to
a Lie group, while the weight spaces of the RotMap layers are Lie groups as well. Figure from [138].

prior over the activities of the top-level features in a deep Boltzmann machine (DBM). This compound HDP-DBM
model learns novel concepts from very few training examples by learning: (i) low-level generic features, (ii) high-level
features that capture correlations among low-level features and, (iii) a category hierarchy for sharing priors over the
high-level features that are typical of different kinds of concepts. Wu and Shao [141] adopted deep belief networks
(DBN) to model the distribution of skeleton joint locations and extract high-level features to represent humans at each
frame in 3D space. Ijjina and Krishna Mohan [142] adopted stacked auto encoder to learn the underlying features of
input skeleton data. Huang et al. [138] incorporated the Lie group structure into a deep learning architecture to learn
more appropriate Lie group features for skeleton based action recognition (see Figure. 16).
5.2. Continuous/Online Motion Recognition
5.2.1. RNN-based Approach
Differently from previous methods that recognize motion from segmented skeleton sequences, Li et al. [137]
proposed a multi-task end-to-end Joint Classification-Regression Recurrent Neural Network to explore the action type
and temporal localization information. They adopted LSTM to capture the complex long-range temporal dynamics,
which avoids the typical sliding window design and thus ensures high computational efficiency. Furthermore, the
subtask of regression optimization provides the ability to forecast the action prior to its occurrence. The framework is
shown in Figure. 15.

6. Multi-modal-based Motion Recognition with Deep Learning
As discussed in previous sections, RGB, depth and skeleton modalities have their own specific properties, and
how to combine the strengths of these modalities with deep learning approach is important. To address this problem,
several methods have been proposed. In general, these methods can be categorized as (i) CNN-based, (ii) RNN-based
24

Figure 17: The deep architecture is composed of five components: (a) Input Preprocessing; (b) 3D CNN (C3D); (c) ConvLSTM; (d) Spatial Pyramid
Pooling and Fully Connected Layers; (e) Multimodal Score Fusion. Figure from [143].

and other-architecture-based approaches for segmented motion recognition and, (iii) RNN-based continuous/online
motion recognition.
6.1. Segmented Motion Recognition
6.1.1. CNN-based Approach
Zhu et al. [144] fused RGB and depth in a pyramidal 3D convolutional network based on C3D [47] for gesture
recognition. They designed pyramid input and pyramid fusion for each modality and late score fusion was adopted
for final recognition. Duan et al. [18] proposed a convolutional two-stream consensus voting network (2SCVN) which
explicitly models both the short-term and long-term structure of the RGB sequences. To alleviate distractions from
background, a 3D depth-saliency ConvNet stream (3DDSN) was aggregated in parallel to identify subtle motion
characteristics. Later score fusion was adopted for final recognition. The methods described so far considered RGB
and depth as separate channels and fused them later. Wang et al. [22] took a different approach and adopted scene
flow to extract features that fused the RGB and depth from the onset. The new representation based on CNN and
named Scene Flow to Action Map (SFAM) was used for motion recognition. Rahmani and Bennamoun [? ] proposed
an end-to-end learning model for action recognition from depth and skeleton data. The proposed model learned
to fuse features from depth and skeletal data, capture the interactions between body-parts and/or interactions with
environmental objects, and model the temporal structure of human actions in an end-to-end learning framework. The
proposed method was made robust to viewpoint changes, by introducing a deep CNN which transfers visual appearance
of human body-parts acquired from different unknown views to a view-invariant space. Similarly, Wang et al. [145]
proposed to represent a depth map sequence as three pairs of structured dynamic images at body, part and joint levels
respectively through bidirectional rank pooling, guided by skeleton data. Differently from previous works that applied
25

Xω

Channel
Transform
Kernels

Y3

Xν
Input SFM

Rank
Pooling
Layer

AlexNet

Y2

Xµ

Y1

Figure 18: The framework of using scene flow for motion recognition. Scene flow vectors are first transformed into Scene Flow Maps (SFM),
and then using Channel Transform Kernels to transform SFM into an analogous RGB space to take advantage of pre-train models over ImageNet.
Figure from [22].

one CNN for each part/joint separately, one pair of structured dynamic images is constructed from depth maps at each
granularity level and serves as the input of a CNN.
6.1.2. RNN-based Approach
For RGB and depth fusion, Pigou et al. [71] directly considered the depth as the fourth channel and CNN was
adopted to extract frame-based appearance features. Temporal convolutions and RNN were combined to capture the
temporal information. Li et al. [146] adopted C3D [47] to extract features separately from RGB and depth modalities,
and used the concatenated for SVM classifier. Zhu et al. [143] presented a gesture recognition method using C3D [47]
and convolutional LSTM (convLSTM) [147] based on depth and RGB modalities (see Figure 17). The major drawback
of traditional LSTM in handling spatio-temporal data is its usage of full connections in input-to-state and state-to-state
transitions in which no spatial information is encoded. The ConvLSTM determines the future state of a certain cell in
the grid by the inputs and past states of its local neighbors. Average score fusion was adopted to fuse the two separate
channel networks for the two modalities. Luo et al. [148] proposed to use a RNN-based encoder-decoder framework
to learn a video representation by predicting a sequence of basic motions described as atomic 3D flows. The learned
representation is then extracted from the generated model to recognize activities.
Shi and Kim [149] fused depth and skeleton in a so-called privileged information (PI)-based RNN (PRNN) that
exploits additional knowledge of skeleton sequences to obtain a better estimate of network parameters from depth map
sequences. A bridging matrix is defined to connect softmax classification loss and regression loss by discovering latent
PI in the refinement step. The whole process is illustrated in Figure 19.
For RGB and skeleton fusion, Mahassein and Todorovic [150] presented a regularization of LSTM learning where
the output of another encoder LSTM (eLSTM) grounded on 3D human-skeleton training data is used as the regularization. This regularization rests on the hypothesis that since videos and skeleton sequences are about human motions
their respective feature representations should be similar. The skeleton sequences, being view-independent and devoid
of background clutter, are expected to facilitate capturing important motion patterns of human-body joints in 3D space.
6.1.3. Other-architecture-based Approach
Shahroudy et al. [151] extracted hand-crafted features which are neither independent nor fully correlated from RGB
and depth, and embedded the input feature into a space of factorized common and modality-specific components. The

26

Figure 19: The framework of PI-based RNNs. It consists of three steps: (1) The pre-training step taking both depth maps and skeleton as input.
An embedded encoder is trained in a standard CNN-RNN pipeline. (2) The trained encoder is used to initialize the learning step. A multi-task
loss is applied to exploit the PI in the regression term as a secondary task. (3) Finally, refinement step aims to discover the latent PI by defining a
bridging matrix, in order to maximize the effectiveness of the PI. The latent PI is utilized to close the gap between different information. The latent
PI, bridging matrix and the network are optimized iteratively in an EM procedure. Figure from [149].

combination of shared and specific components in input features can be very complex and highly non-linear. In order
to disentangle them, they stacked layers of non-linear auto encoder-based component factorization to form a deep
shared-specific analysis network.
In a RGB, depth and skeleton fusion method, Wu et al. [152] adopted Gaussian-Bernouilli Deep Belief Network(DBN) to extract high-level skeletal joint features and the learned representation is used to estimate the emission
probability needed to infer gesture sequences. A 3D Convolutional Neural Network (3DCNN) was used to extract
features from 2D multiple channel inputs such as depth and RGB images stacked along the 1D temporal domain. In
addition, intermediate and late fusion strategies were investigated in combination with the temporal modeling. The
result of both mechanisms indicates that multiple-channel fusion can outperform individual modules.
6.2. Continuous/Online Motion Recognition
6.2.1. RNN-based Approach
Chai et al. [153] proposed to fuse RGB and depth in a two-stream RNN (2S-RNN) for gesture recognition. They
designed a fusion layer for depth and RGB before the LSTM layer. [154] presented an algorithm for joint segmentation
and classification of dynamic hand gestures from continuous video streams. They proposed a network that employs
a recurrent C3D with connectionist temporal classification (CTC) [155]. They trained a separate network for each
modality and averaged their scores for final recognition. Beside RGB and depth modalities, they also adopted stereoIR modality in their work.

27

7. Discussion
We presented a comprehensive overview of RGB-D based motion recognition using deep learning. We defined
a taxonomy covering two groups: segmented and continuous/online motion recognition, with four categories in each
group based on the adopted modalities. From the viewpoint of encoding spatio-temporal-structural information contained in the video sequences, CNN, RNN and other networks adopted for motion recognition are discussed in each
category. In subsequent sections, the relative performance of the different methods on several commonly used RGB-D
datasets are analysed, and from the comparisons we highlight some challenges. The discussion on performance and
challenges then provides a basis for outlining potential future research directions.
7.1. Performance Analysis of the Current Methods
In this section, we compare the accuracy of different methods using several commonly used datasets, including
CMU Mocap, HDM05, MSR-Action3D, MSRC-12, MSRDailyActivity3D, UTKinect, G3D, SBU Kinect Interaction,
Berkeley MHAD, Northwestern-UCLA Multiview Action3D, ChaLearn LAP IsoGD, NTU RGB+D, ChaLearn2014,
ChaLearn LAP ConGD, and PKU-MMD. These datasets cover motion capture sensor system, structured light cameras (Kinect v1) and ToF cameras (Kinect v2). The last three datasets are continuous datasets while the others are
segmented datasets. The performance is evaluated using accuracy for segmented motion recognition, and Jaccard
Index is added as another criteria for continuous motion recognition. The accuracy is calculated as the proportion
of accurately labelled samples. The Jaccard index measures the average relative overlap between true and predicted
sequences of frames for a given gesture/action. For a sequence s, let Gs,i and Ps,i be binary indicator vectors for
which 1-values correspond to frames in which the ith gesture/action label is being performed. The Jaccard Index for
the ith class is defined for the sequence s as:

Js,i =

T
Gs,i Ps,i
S
,
Gs,i Ps,i

(1)

where Gs,i is the ground truth of the ith gesture/action label in sequence s, and Ps,i is the prediction for the ith label
in sequence s. When Gs,i and Ps,i are empty, J(s,i) is defined to be 0. Then for the sequence s with ls true labels, the
Jaccard Index Js is calculated as:

Js =

L
1X
Js,i .
ls i=1

(2)

For all test sequences S = s1 , ..., sn with n gestures/actions, the mean Jaccard Index JS is used as the evaluation
criteria and calculated as:

n

JS =

1X
Js .
n j=1 j

(3)

The detailed comparison of different methods is presented in following Table 2. From the Table we can see
that there is no single approach that is able to produce the best performance over all datasets. Generally speaking,
methods using multi-modal information can have better performance than their single modality counterpart due to the

28

complementary properties of the three different modalities. On some datasets, such as NTU RGB+D dataset, current
results suggest that CNN-based methods tend to be better than RNN-based methods. This is probably due to fact that
CNN-based methods includes human empirical knowledge in the coding process, and could take advantage of pretrained models over large image set, such as ImageNet. The combination of CNN and RNN seems to be a good choice
for motion recognition, for instance, the C3D+ConvLSTM [143] method achieved promising results on ChaLearn
LAP IsoGD dataset. For continuous motion recognition, RNN-based methods tend to achieve good results.
7.2. Challenges
The advent of low-cost RGB-D sensors that have access to extra depth and skeleton data, has motivated the significant development of human motion recognition. Promising results have been achieved with deep learning approaches [115, 133, 131], on several constrained simple datasets, such as MSR-Action3D, Berkeley MHAD and SBU
Kinect Interaction. Despite this success, results are far from satisfactory on some large complex datasets, such as
ChaLearn LAP IsoGD and NTU RGB+D datasets and especially the continuous/online datasets. In fact, it is still very
difficult to build a practical intelligent recognition system. Such goal poses several challenges:
Encoding temporal information. As discussed, there are several methods to encode temporal information. We
can use CNN to extract frame-based features and then conduct temporal fusion [45], or adopt 3D filter and 3D pooling
layers to learn motion features [47], or use optical/scene flow to extract motion information [56, 22], or encode the
video into images [51, 115, 21], or use RNN/LSTM to model the temporal dependences [70, 127, 20]. However,
all these approaches have their drawbacks. Temporal fusion method tends to neglect the temporal order; 3D filters
and 3D pooling filters have a very rigid temporal structure and they only accept a predefined number of frames as
input which is always short; optical/scene flow methods are computationally expensive; sequence to images methods
inevitably loses temporal information during encoding; the weight sharing mechanism of RNN/LSTM methods make
the sequence matching imprecise, but rather approximated, so an appropriate distance function must be used to predict
the match probability. In fact, there is still no perfect method for temporal encoding, and how to model temporal
information is a big challenge.
Small training data. Most of available deep learning methods rely on large labeled training data [45, 47]. However, in practical scenarios, obtaining large labeled training data is costly and laborious, even impossible, especially in
medical-related applications. It has been shown that fine-tuning motion-based networks with spatial data (ImageNet)
is more effective than training from scratch [56, 21, 51, 22]. Strategies for data augmentation are also commonly
used [115]. Likewise, training mechanisms to avoid overfitting and control learning rate have also been studied [156].
However, it is still a challenge to effectively train deep networks from small training data.
Viewpoint variation and occlusion. Viewpoint variation may cause significantly different appearance of the same
action, and occlusion would “crash” the skeleton data. Occlusion includes inter-occlusion caused by other subjects or
objects, and self-occlusion created by the object/subject itself. Most of available datasets require subjects to perform
actions in a visible and restricted view to avoid occlusion, and this results in limited view data collection and less

29

Table 2: Performance comparison among different methods on commonly used RGB-D datasets. Notation: D:Depth, S:Skeleton, Acc:Accuracy,
JI:Jaccard Index, cs:cross-subject setting, cv:cross-view setting. Without specific notation, accuracy is used for the metric.

Dataset

Reference

Modality

Method

Fusion

Metric

Method
CMU Mocap

HDM05

[130]

Skeleton

Co-occurrence+LSTM

None

81.04%

[125]

Skeleton

Clips+CNN+MTLN

None

88.30%

[138]

Skeleton

None

75.78%

Deep Learning
on Lie Group

MSR-Action3D

[127]

Skeleton

HBRNN-L

None

96.92%

[130]

Skeleton

Co-occurrence+LSTM

None

97.25%

[113]

Depth

3DCNN

None

84.07%

[129]

Skeleton

dRNN

None

92.03%

[127]

Skeleton

HBRNN-L

None

94.49%

[149]

Depth+

PRNN

Skeleton

MSRC-12

MSRDaily

Side

94.90%

Information

[115]

Depth

WHDMM+CNN

None

100.00%

[21]

Skeleton

JTM+CNN

None

93.12%

[121]

Skeleton

SOS+CNN

None

94.27%

[123]

Skeleton

Enhanced Visualization+CNN

None

96.62%

[115]

Depth

WHDMM+CNN

None

85.00%

[148]

Depth

None

86.90%

Activity3D
Unsupervised
+ConvLSTM
[151]

RGB+

DSSCA-SSLM

Depth
UTKinect

G3D

Hierarchical

97.50%

Fusion

[113]

Depth

3DCNN

None

82.00%

[115]

Depth

WHDMM+CNN

None

90.91%

[133]

Skeleton

JL d+RNN

None

95.96%

[134]

Skeleton

Ensemble TS-LSTM

Ensemble

96.97%

[131]

Skeleton

ST-LSTM+Trust Gate

None

97.00%

[138]

Skeleton

None

89.10%

Deep Learning
on Lie Group

[21]

Skeleton

JTM+CNN

None

94.24%

[121]

Skeleton

SOS+CNN

None

95.45%

30

SBU Kinect

[149]

Interaction

Depth+

PRNN

Skeleton
[130]

Skeleton

Side

89.20%

Information
Co-occurrence

None

90.41%

+LSTM

Berkeley MHAD

Multiview

[132]

Skeleton

STA-LSTM

None

91.51%

[131]

Skeleton

ST-LSTM+Trust Gate

None

93.30%

[124]

Skeleton

SkeletonNet(CNN)

None

93.47%

[125]

Skeleton

Clips+CNN+MTLN

None

93.57%

[133]

Skeleton

JL d+RNN

None

99.02%

[142]

Skeleton

Stacked Autoencoder

None

98.03%

[127]

Skeleton

HBRNN-L

None

100.00%

[133]

Skeleton

JL d+RNN

None

100.00%

[120]

Skeleton

Skeleton Matrix+CNN

None

100.00%

[131]

Skeleton

ST-LSTM+Trust Gate

None

100.00%

[117]

Depth

Fitting model+CNN

None

92.00%

[123]

Skeleton

None

92.61%

Sceneflow+CNN

Early Fusion

36.27%

DynamicImages+CNN

None

39.23%

Pyramidal C3D

Score Fusion

45.02%

2SCVN-3DDSN

Score Fusion

49.17%

C3D

Score Fusion

49.20%

C3D+ConvLSTM

Score Fusion

51.02%

None

56.00%(cs)

Action3D
Enhanced Visualization
+CNN
ChaLearn LAP

[22]

IsoGD

RGB+
Depth

[19]
[144]

Depth
RGB+
Depth

[18]

RGB+
Depth

[157]

RGB+
Depth

[143]

RGB+
Depth

NTU RGB+D

[148]

RGB

Unsupervised
+ConvLSTM

[138]

Skeleton

Deep Learning

None

on Lie Group
[34]

Skeleton

2Layer P-LSTM

61.37%(cs)
66.95%(cv)

None

62.93%(cs)
70.27%(cv)

[148]

Depth

Unsupervised

None

66.20%(cs)

+ConvLSTM
[131]

Skeleton

31
ST-LSTM+Trust Gate

None

69.20%(cs)
77.70%(cv)
70.26%(cs)

[134]

Skeleton

Ensemble TS-LSTM

Ensemble

74.60%(cs)
81.25%(cv)

[151]

RGB+

DSSCA-SSLM

Hierarchical Fusion

SkeletonNet(CNN)

None

74.86%(cs)

Depth
[124]

Skeleton

75.94%(cs)
81.16%(cv)

[122]

Skeleton

JDM+CNN

None

76.20%(cs)
82.30%(cv)

[125]

Skeleton

Clips+CNN+MTLN

None

79.57%(cs)
84.83%(cv)

[123]

Skeleton

Enhanced Visualization

None

+CNN

ChaLearn2014

80.03%(cs)
87.21%(cv)

[158]

RGB

Chained Multi-stream

Markov chain

80.80%(cs)

[120]

Skeleton

Skeleton Matrix+CNN

None

91.16%

[71]

RGB

Temp Conv+RNN,LSTM

None

94.49%
0.906(JI)

[154]

RGB+

CTC+RNN

Score Fusion

Depth

98.20%
0.980(JI)

RGB+
[152]

Depth+

DDNN

Late Fusion

0.809(JI)

IDMM+CNN

None

0.240(JI)

RNN

Score Fusion

0.266(JI)

C3D

None

0.343(JI)

Skeleton
ChaLearn LAP

[118]

Depth

ConGD
[153]

RGB+
Depth

PKU-MMD

[159]

RGB

[132]

S

SlidingWindow

None

+STA-LSTM
[137]

S

JCR-RNN

0.427(JI)(cs)
0.468(JI)(cv)

None

0.479(JI)(cs)

0.728(JI)(cv)
occlusion. However, occlusion is inevitable in practical scenarios, especially for interactions. This makes it challenging to isolate individuals in overlapping area and extract features of a unique person; leading to the ineffectiveness of
many of available approaches [127, 34, 122]. Possible solutions to handle viewpoint variation and occlusion include
the use of multi-sensor systems [31, 32, 34, 36]. The multi-camera systems is able to generate multi-view data, but the
drawback is the requirement of synchronization and feature/recognition fusion among different views. This usually
increases processing complexity and computation cost. Several methods have been proposed to handle the viewpoint

32

variation and occlusion. Wang et al. [114] proposed to rotate the depth data in 3D point clouds through different angles
to deal with viewpoint invariance; spherical coordinate system corresponding to body center was developed to achieve
view-independent motion recognition [138]. However, these methods become less effective when occlusion occurs.
How to effectively handle occlusion using deep learning methods is a new challenge.
Execution rate variation and repetition. The execution rate may vary due to the different performing styles and
states of individuals. The varying rate results in different frames for the same motion. Repetition also bring about
this issue. The global encoding methods [121, 124, 123] would become less effective due to the repetition. The
commonly used methods to handle this problem is up/down sampling [130, 133, 122]. However, sampling methods
would inevitably bring redundant or loss of useful information. Effective handling of this problem remains a challenge.
Cross-datasets. Many research works have been carried out to recognize human actions from RGB-D video
clips. To learn an effective action classifier, most of the previous approaches rely on enough training labels. When
being required to recognize the action in a different dataset, these approaches have to re-train the model using new
labels. However, labeling video sequences is a very tedious and time-consuming task, especially when detailed spatial
locations and time durations are required. Even though some works have studied this topic [160, 161, 162], they
are all based on hand-crafted features, and the results are far from satisfactory due to the large distribution variances
between different datasets, including different scenarios, different modalities, different views, different persons, and
even different actions. How to deal with cross-datasets RGB-D motion recognition is a big challenge.
Online motion recognition. Most of available methods rely on segmented data, and their capability for online
recognition is quite limited. Even though continuous motion recognition is one improved version where the videos are
untrimmed, it still assumes that all the videos are available before processing. Thus, proposal-based methods [95, 97]
can be adopted for offline processing. Differently from continuous motion recognition, online motion recognition aims
to receive continuous streams of unprocessed visual data and recognize actions from an unsegmented stream of data
in a continuous manner. So far two main approaches can be identified for online recognition, sliding window-based
and RNN-based. Sliding window-based methods [36] are simple extension of segmented-based action recognition
methods. They often consider the temporal coherence within the window for prediction and the window-based predictions are further fused to achieve online recognition. However, the performance of these methods are sensitive
to the window size which depends on actions and is hard to set. Either too large or too small a window size could
lead to significant drop in recognition. For RNN-based methods [154, 137], even though promising results have been
achieved, it is still far from satisfactory in terms of performance. How to design effective practical online recognition
system is a big challenge.
Action prediction. We are faced with numerous situations in which we must predict what actions other people
are about to do in the near future. Predicting future actions before they are actually executed is a critical ingredient
for enabling us to effectively interact with other humans on a daily basis [163, 164, 165, 166, 167]. There are mainly
two challenges for this task: first, we need to capture the subtle details inherent in human movements that may imply
a future action; second, predictions usually should be carried out as quickly as possible in the social world, when
limited prior observations are available. Predicting the action of a person before it is actually executed has a wide

33

range of applications in autonomous robots, surveillance and health care. How to develop effective algorithms for
action prediction is really challenging.
7.3. Future Research Directions
The discussion on the challenges faced by available methods allows us to outline several future research directions
for the development of deep learning methods for motion recognition. While the list is not exhaustive, they point at
research activities that may advance the field.
Hybrid networks. Most of previous methods adopted one type of neural networks for motion recognition. As
discussed, there is no perfect solution for temporal encoding using single networks. Even though available works
such as C3D+ConvLSTM [143] used two types of networks, the cascaded connection makes them dependent on each
other during training. How to cooperatively train different kinds of networks would be a good research direction; for
example, using the output of CNN to regularize RNN training in parallel.
Simultaneous exploitation of spatial-temporal-structural information. A video sequence has three important
inherent properties that should be considered for motion analysis: spatial information, temporal information and structural information. Several previous methods tend to exploit the spatio-temporal information for motion recognition,
however, structural information contained in the video is rarely explicitly mined. Concurrent mining of these three
kinds of information with deep learning would be an interesting topic in the future [76].
Fusion of multiple modalities. While significant progress has been achieved by singly using RGB, skeleton or
depth modality, effective deep networks for fusion of multi-modal data would be a promising direction. For example,
methods such as SFAM [22] and PRNN [149] have pioneered the research in this direction. The work SFAM [22]
proposed to extract scene flow for motion analysis. The strategy of fusing the RGB and depth modalities at the
outset allowed the capture of rich 3D motion information. In PRNN [149] the concept of privileged information (side
information) was introduced for deep networks training and showed some promise. Zolfaghari et al. [158] proposed
the integration of different modalities via a Markov chain, which leads to a sequential refinement of action labels. So
far, most methods considered the three modalities as separate channels and fused them at a later or scoring stage using
different fusion methods without cooperatively exploiting their complementary properties. Cooperative training using
different modalities would be a promising research area.
Large-scale datasets. With the development of data-hungry deep learning approach, there is demand for large
scale RGB-D datasets. Even though there are several large datasets, such as NTU RGB+D Dataset [34] and ChaLearn
LAP IsoGD Dataset [33], they are focused on specific tasks. Various large-scale RGB-D datasets are needed to
facilitate research in this field. For instance, large-scale fine-grained RGB-D motion recognition datasets and largescale occlusion-based RGB-D motion recognition datasets are urgently needed.
Zero/One-shot learning. As discussed, it is not always easy to collect large scale labeled data. Learning from
a few examples remains a key challenge in machine learning. Despite recent advances in important domains such
as vision and language, the standard supervised deep learning paradigm does not offer a satisfactory solution for
learning new concepts rapidly from little data. How to adopt deep learning methods for zero/one shot RGB-D-based

34

motion recognition would be an interesting research direction. Zero/one-shot learning is about being able to recognize
gesture/action classes that are never seen or only one training sample per class before. This type of recognition
should carry embedded information universal to all other gestures/actions. In the past few years, there are some works
on zero/one-shot learning. For example, Wan et al. [168] proposed the novel spatial-temporal features for one-shot
learning gesture recognition and have got promising performances on Chalearn Gesture Dataset CGD) [169]. For zeroshot learning, Madapana and Wachs [170] proposed a new paradigm based on adaptive learning which it is possible
to determine the amount of transfer learning carried out by the algorithm and how much knowledge is acquired for
a new gesture observation. However, the mentioned works used traditional methods (such as bag of visual words
model [171]). Mettes et al. [172] proposed a spatial-aware object embedding for zero-shot action localization and
classification. The spatial-aware embedding generate action tubes by incorporating word embeddings, box locations
for actors and objects, as well as their spatial relations. However, how to effectively adopt deep learning methods for
zero/one shot RGB-D based motion recognition would be still an interesting research direction especially when using
only very few training samples.
Outdoor practical scenarios. Although lots of RGB-D datasets have been collected during the last few years,
there is a big gap between the collected datasets and wild environment due to constrained environment setting and insufficient categories and samples. For example, most available datasets do not involve much occlusion cases probably
due to the collapse of skeleton dataset in case of occlusion. However, in practical scenarios, occlusion is inevitable.
How to recover or find cues from multi-modal data for such recognition tasks would be an interesting research direction. Besides, with the development of depth sensors, further distances could be captured, and recognition in outdoor
practical scenarios will gain the attention of researchers.
Unsupervised learning/Self-learning. Collecting labeled datasets are time-consuming and costly, hence learning
from unsupervised video data is required. Mobile robots mounted with RGB-D cameras need to continuously learn
from the environment and without human intervention. How to automatically learn from the unlabeled data stream
to improve the learning capability of deep networks would be a fruitful and useful research direction. Generative
Adversarial Net (GAN) [173] has got much processes recently in image generation task, such as face generation,
text-to-image task. Besides, it also can be used for recognition task. For example, Luan et al. [174] proposed a Disentangled Representation learning Generative Adversarial Networks (DR-GAN) for pose-invariant face recognition.
Therefore, we believe the GAN-based techniques also can be used for action/gesture recognition, which is a great
exciting direction for research. Carl et al. [175] proposed a generative adversarial network for video with spatialtemporal convolutional architecture that untangles the scene’s foreground from backgrounds. This is an initial work
to capitalize on large amounts of unlabeled video in order to learn a model of scene dynamic for both video recognition tasks (e.g. action classification) and video generation tasks (e.g. future prediction). Increasing research will be
reported in the coming years on GAN-based methods for video-based recognition.
Online motion recognition and prediction. Online motion recognition and prediction is required in practical
applications, and arguably this is the final goal of motion recognition systems. Differently from segmented recognition,
online motion recognition requires the analysis of human behavior in a continuous manner, and prediction aims to

35

recognize or anticipate actions that would happen. How to design effective online recognition and prediction systems
with deep learning methods has attracted some attention. For example, Vondrick et al. [176] introduced a framework
that capitalizes on temporal structure in unlabeled video to learn to anticipate human actions and objects based on
CNN, and it is likely to emerge as an active research area.

8. Conclusion
This paper presents a comprehensive survey of RGB-D based motion recognition using deep learning. We provide
a brief overview of existing commonly used datasets and pointed at surveys that focused mainly on datasets. The
available methods are grouped into four categories according to the modality: RGB-based, depth-based, skeletonbased and multi-modal-based. The three modalities have their own specific features and lead to different choices of
deep learning methods to take advantages of their properties. Spatial, temporal and structural information inherent
in a video sequence is defined, and from the viewpoint of spatio-temporal-structural encoding, we analyse the pros
and cons of available methods. Based on the insights drawn from the survey, several potential research directions are
described, indicating the numerous opportunities in this field despite the advances achieved to date.

Acknowledgment
Jun Wan is partially supported by the National Natural Science Foundation of China [61502491]. Sergio Escalera
is partially supported by Spanish project [TIN2016-74946-P] (MINECO/FEDER, UE) and CERCA Programme /
Generalitat de Catalunya.

References
References
[1] J. Aggarwal, Q. Cai, Human motion analysis: A review, in: Computer Vision and Image Understanding, 1999.
[2] L. Wang, W. Hu, T. Tan, Recent developments in human motion analysis, Pattern recognition 36 (3) (2003)
585–601.
[3] P. Turaga, R. Chellappa, V. S. Subrahmanian, O. Udrea, Machine recognition of human activities: A survey,
IEEE Transactions on Circuits and Systems for Video Technology 18 (11) (2008) 1473–1488.
[4] R. Poppe, A survey on vision-based human action recognition, Image and vision computing 28 (6) (2010)
976–990.
[5] G. Guo, A. Lai, A survey on still image based human action recognition, Pattern Recognition 47 (10) (2014)
3343–3361.
[6] F. Zhu, L. Shao, J. Xie, Y. Fang, From handcrafted to learned representations for human action recognition: a
survey, Image and Vision Computing 55 (2016) 42–52.
36

[7] S. Herath, M. Harandi, F. Porikli, Going deeper into action recognition: A survey, Image and Vision Computing
60 (2017) 4–21.
[8] E. Hugo, C. Albert, E. Sergio, B. Marco, A. Maryam, B. Xavier, P.-L. Victor, e. G. Isabell, K. Shohreh, A
survey on deep learning based approaches for action and gesture recognition in image sequences, in: FG, 2017.
[9] J. Shotton, A. Fitzgibbon, M. Cook, T. Sharp, M. Finocchio, R. Moore, A. Kipman, A. Blake, Real-time human
pose recognition in parts from single depth images, in: Proc. IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), 2011, pp. 1297–1304.
[10] L. Chen, H. Wei, J. Ferryman, A survey of human motion analysis using depth imagery, Pattern Recognition
Letters 34 (15) (2013) 1995–2006.
[11] M. Ye, Q. Zhang, L. Wang, J. Zhu, R. Yang, J. Gall, A survey on human motion analysis from depth data, in:
Time-of-Flight and Depth Imaging. Sensors, Algorithms, and Applications, 2013, pp. 149–187.
[12] J. K. Aggarwal, L. Xia, Human activity recognition from 3d data: A review, Pattern Recognition Letters 48
(2014) 70–80.
[13] H. Cheng, L. Yang, Z. Liu, Survey on 3d hand gesture recognition, IEEE Transactions on Circuits and Systems
for Video Technology 26 (9) (2016) 1659–1673.
[14] J. Zhang, W. Li, P. O. Ogunbona, P. Wang, C. Tang, RGB-D-based action recognition datasets: A survey, Pattern
Recognition 60 (2016) 86–105.
[15] S. Escalera, V. Athitsos, I. Guyon, Challenges in multimodal gesture recognition, Journal of Machine Learning
Research 17 (72) (2016) 1–54.
[16] L. L. Presti, M. La Cascia, 3d skeleton-based human action classification: A survey, Pattern Recognition 53
(2016) 130–147.
[17] F. Han, B. Reily, W. Hoff, H. Zhang, Space-time representation of people based on 3d skeletal data: A review,
Computer Vision and Image Understanding.
[18] J. Duan, S. Zhou, J. Wan, X. Guo, S. Z. Li, Multi-modality fusion based on consensus-voting and 3d convolution
for isolated gesture recognition, arXiv preprint arXiv:1611.06689.
[19] P. Wang, W. Li, S. Liu, Z. Gao, C. Tang, P. Ogunbona, Large-scale isolated gesture recognition using convolutional neural networks, in: Pattern Recognition (ICPR), 2016 23rd International Conference on, IEEE, 2016,
pp. 7–12.
[20] J. Liu, G. Wang, P. Hu, L.-Y. Duan, A. C.Kot, Global context-aware attention lstm networks for 3d action
recognition, in: CVPR, 2017.

37

[21] P. Wang, Z. Li, Y. Hou, W. Li, Action recognition based on joint trajectory maps using convolutional neural
networks, in: ACM MM, 2016.
[22] P. Wang, W. Li, Z. Gao, Y. Zhang, C. Tang, P. Ogunbona, Scene flow to action map: A new representation for
rgb-d based action recognition with convolutional neural networks, in: CVPR, 2017.
[23] CMU Graphics Lab Motion Capture Database, http://mocap.cs.cmu.edu/ (2001).
[24] M. Müller, T. Röder, M. Clausen, B. Eberhardt, B. Krüger, A. Weber, Documentation mocap database hdm05,
Tech. Rep. CG-2007-2, Universität Bonn (June 2007).
[25] W. Li, Z. Zhang, Z. Liu, Action recognition based on a bag of 3D points, in: CVPRW, 2010.
[26] S. Fothergill, H. M. Mentis, S. Nowozin, P. Kohli, Instructing people for training gestural interactive systems,
in: ACM Conference on Computer-Human Interaction (ACM HCI), 2012.
[27] J. Wang, Z. Liu, Y. Wu, J. Yuan, Mining actionlet ensemble for action recognition with depth cameras, in:
CVPR, 2012.
[28] L. Xia, C.-C. Chen, J. Aggarwal, View invariant human action recognition using histograms of 3D joints, in:
CVPRW, 2012.
[29] V. Bloom, D. Makris, V. Argyriou, G3D: A gaming action dataset and real time action recognition evaluation
framework, in: CVPRW, 2012.
[30] K. Yun, J. Honorio, D. Chattopadhyay, T. L. Berg, D. Samaras, Two-person interaction detection using
body-pose features and multiple instance learning, in: Computer Vision and Pattern Recognition Workshops
(CVPRW), 2012 IEEE Computer Society Conference on, IEEE, 2012, pp. 28–35.
[31] F. Ofli, R. Chaudhry, G. Kurillo, R. Vidal, R. Bajcsy, Berkeley MHAD: A comprehensive multimodal human
action database, in: Proc. IEEE Workshop on Applications of Computer Vision (WACV), 2013, pp. 53–60.
[32] J. Wang, X. Nie, Y. Xia, Y. Wu, S.-C. Zhu, Cross-view action modeling, learning and recognition, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 2649–2656.
[33] J. Wan, S. Z. Li, Y. Zhao, S. Zhou, I. Guyon, S. Escalera, Chalearn looking at people RGB-D isolated and
continuous datasets for gesture recognition, in: CVPRW, 2016.
[34] A. Shahroudy, J. Liu, T.-T. Ng, G. Wang, NTU RGB+ D: A large scale dataset for 3D human activity analysis,
in: CVPR, 2016.
[35] S. Escalera, X. Baró, J. Gonzalez, M. A. Bautista, M. Madadi, M. Reyes, V. Ponce-López, H. J. Escalante,
J. Shotton, I. Guyon, Chalearn looking at people challenge 2014: Dataset and results, in: Workshop at the
European Conference on Computer Vision, Springer, 2014, pp. 459–473.

38

[36] L. Chunhui, H. Yueyu, L. Yanghao, S. Sijie, L. Jiaying, Pku-mmd: A large scale benchmark for continuous
multi-modal human action understanding, arXiv preprint arXiv:1703.07475.
[37] F. C. Heilbron, V. Escorcia, B. Ghanem, J. C. Niebles, Activitynet: A large-scale video benchmark for human
activity understanding, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
2015, pp. 961–970.
[38] R. De Geest, E. Gavves, A. Ghodrati, Z. Li, C. Snoek, T. Tuytelaars, Online action detection, in: ECCV, 2016,
pp. 269–284.
[39] S. Yeung, O. Russakovsky, N. Jin, M. Andriluka, G. Mori, L. Fei-Fei, Every moment counts: Dense detailed
labeling of actions in complex videos, International Journal of Computer Vision.
[40] A. Gorban, H. Idrees, Y. Jiang, A. R. Zamir, I. Laptev, M. Shah, R. Sukthankar, Thumos challenge: Action
recognition with a large number of classes, in: CVPR workshop, 2015.
[41] I. Goodfellow,

Y. Bengio,

A. Courville,

Deep Learning,

MIT Press,

2016,

http://www.

deeplearningbook.org.
[42] A. Krizhevsky, I. Sutskever, G. E. Hinton, Imagenet classification with deep convolutional neural networks, in:
Proc. Annual Conference on Neural Information Processing Systems (NIPS), 2012, pp. 1106–1114.
[43] K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition, arXiv preprint
arXiv:1409.1556.
[44] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition, 2016, pp. 770–778.
[45] A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, L. Fei-Fei, Large-scale video classification
with convolutional neural networks, in: Proc. IEEE Conference on Computer Vision and Pattern Recognition
(CVPR), 2014, pp. 1725–1732.
[46] J. Ng, M. Hausknecht, S. Vijayanarasimhan, O. Vinyals, R. Monga, G. Toderici, Beyond short snippets: Deep
networks for video classification, in: CVPR, 2015.
[47] D. Tran, L. Bourdev, R. Fergus, L. Torresani, M. Paluri, Learning spatiotemporal features with 3D convolutional
networks, in: ICCV, 2015.
[48] S. Ji, W. Xu, M. Yang, K. Yu, 3D convolutional neural networks for human action recognition, TPAMI 35 (1)
(2013) 221–231.
[49] G. Varol, I. Laptev, C. Schmid, Long-term temporal convolutions for action recognition, arXiv preprint
arXiv:1604.04494.

39

[50] L. Sun, K. Jia, D.-Y. Yeung, B. E. Shi, Human action recognition using factorized spatio-temporal convolutional
networks, in: Proceedings of the IEEE International Conference on Computer Vision, 2015, pp. 4597–4605.
[51] H. Bilen, B. Fernando, E. Gavves, A. Vedaldi, S. Gould, Dynamic image networks for action recognition, in:
CVPR, 2016.
[52] B. Fernando, S. Gavves, O. Mogrovejo, J. Antonio, A. Ghodrati, T. Tuytelaars, Rank pooling for action recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence.
[53] B. Fernando, S. Gould, Learning end-to-end video classification with rank-pooling, in: ICML, 2016.
[54] B. Fernando, P. Anderson, M. Hutter, S. Gould, Discriminative hierarchical rank pooling for activity recognition, in: CVPR, 2016.
[55] A. Cherian, B. Fernando, M. Harandi, S. Gould, Generalized rank pooling for activity recognition, in: CVPR,
2017.
[56] K. Simonyan, A. Zisserman, Two-stream convolutional networks for action recognition in videos, in: NIPS,
2014.
[57] L. Wang, Y. Qiao, X. Tang, Action recognition with trajectory-pooled deep-convolutional descriptors, in:
CVPR, 2015, pp. 4305–4314.
[58] H. Wang, A. Kläser, C. Schmid, C.-L. Liu, Dense trajectories and motion boundary descriptors for action
recognition, International Journal of Computer Vision 103 (1) (2013) 60–79.
[59] L. Wang, Y. Xiong, Z. Wang, Y. Qiao, D. Lin, X. Tang, L. Van Gool, Temporal segment networks: towards
good practices for deep action recognition, in: European Conference on Computer Vision, 2016, pp. 20–36.
[60] B. Zhang, L. Wang, Z. Wang, Y. Qiao, H. Wang, Real-time action recognition with enhanced motion vector
cnns, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 2718–
2726.
[61] Y. Wang, J. Song, L. Wang, L. Van Gool, O. Hilliges, Two-stream sr-cnns for action recognition in videos,
BMVC, 2016.
[62] G. Chéron, I. Laptev, C. Schmid, P-cnn: Pose-based cnn features for action recognition, in: ICCV, 2015, pp.
3218–3226.
[63] X. Wang, A. Farhadi, A. Gupta, Actions˜ transformations, in: Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition, 2016, pp. 2658–2667.
[64] W. Zhu, J. Hu, G. Sun, X. Cao, Y. Qiao, A key volume mining deep framework for action recognition, in:
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 1991–1999.

40

[65] C. Feichtenhofer, A. Pinz, R. Wildes, Spatiotemporal residual networks for video action recognition, in: Advances in Neural Information Processing Systems, 2016, pp. 3468–3476.
[66] C. Lea, A. Reiter, R. Vidal, G. D. Hager, Segmental spatiotemporal cnns for fine-grained action segmentation,
in: European Conference on Computer Vision, Springer, 2016, pp. 36–52.
[67] A. F. Bobick, J. W. Davis, The recognition of human movement using temporal templates, IEEE Transactions
on pattern analysis and machine intelligence 23 (3) (2001) 257–267.
[68] C. Feichtenhofer, A. Pinz, A. Zisserman, Convolutional two-stream network fusion for video action recognition,
in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 1933–1941.
[69] M. Baccouche, F. Mamalet, C. Wolf, C. Garcia, A. Baskurt, Sequential deep learning for human action recognition, in: International Workshop on Human Behavior Understanding, Springer, 2011, pp. 29–39.
[70] J. Donahue, L. Anne Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, T. Darrell, Longterm recurrent convolutional networks for visual recognition and description, in: CVPR, 2015.
[71] L. Pigou, A. Van Den Oord, S. Dieleman, M. Van Herreweghe, J. Dambre, Beyond temporal pooling: Recurrence and temporal convolutions for gesture recognition in video, International Journal of Computer Vision
(2016) 1–10.
[72] S. Sharma, R. Kiros, R. Salakhutdinov, Action recognition using visual attention, ICLRW.
[73] W. Du, Y. Wang, Y. Qiao, Rpan: An end-to-end recurrent pose-attention network for action recognition in
videos, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 3725–
3734.
[74] J. Sánchez, F. Perronnin, T. Mensink, J. Verbeek, Image classification with the fisher vector: Theory and practice, International journal of computer vision 105 (3) (2013) 222–245.
[75] G. Lev, G. Sadeh, B. Klein, L. Wolf, Rnn fisher vectors for action recognition and image annotation, in: European Conference on Computer Vision, Springer, 2016, pp. 833–850.
[76] A. Jain, A. R. Zamir, S. Savarese, A. Saxena, Structural-rnn: Deep learning on spatio-temporal graphs, in:
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 5308–5317.
[77] L. Sun, K. Jia, K. Chen, D. Y. Yeung, B. E. Shi, S. Savarese, Lattice long short-term memory for human action
recognition, arXiv preprint arXiv:1708.03958.
[78] Y. Shi, Y. Tian, Y. Wang, W. Zeng, T. Huang, Learning long-term dependencies for action recognition with a
biologically-inspired deep network, in: ICCV, 2017, pp. 716–725.
[79] V. Escorcia, F. C. Heilbron, J. C. Niebles, B. Ghanem, Daps: Deep action proposals for action understanding,
in: European Conference on Computer Vision, Springer, 2016, pp. 768–784.
41

[80] N. Srivastava, E. Mansimov, R. Salakhudinov, Unsupervised learning of video representations using lstms, in:
ICML, 2015, pp. 843–852.
[81] H. Jhuang, T. Serre, L. Wolf, T. Poggio, A biologically inspired system for action recognition, in: Proc. IEEE
11th International Conference on Computer Vision, 2007, pp. 1–8.
[82] B. Chen, Deep learning of invariant spatio-temporal features from video, Ph.D. thesis, University of British
Columbia (2010).
[83] G. W. Taylor, R. Fergus, Y. LeCun, C. Bregler, Convolutional learning of spatio-temporal features, in: Proc.
European Conference on Computer Vision (ECCV), 2010, pp. 140–153.
[84] R. Memisevic, G. Hinton, Unsupervised learning of image transformations, in: Computer Vision and Pattern
Recognition, IEEE Conference on, IEEE, 2007, pp. 1–8.
[85] Q. V. Le, W. Y. Zou, S. Y. Yeung, A. Y. Ng, Learning hierarchical invariant spatio-temporal features for action
recognition with independent subspace analysis, in: Computer Vision and Pattern Recognition (CVPR), 2011
IEEE Conference on, IEEE, 2011, pp. 3361–3368.
[86] F. J. Theis, Towards a general independent subspace analysis, in: Advances in Neural Information Processing
Systems, 2007, pp. 1361–1368.
[87] X. Yan, H. Chang, S. Shan, X. Chen, Modeling video dynamics with deep dynencoder, in: European Conference
on Computer Vision, Springer, 2014, pp. 215–230.
[88] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, Generative adversarial nets, in: Advances in neural information processing systems, 2014, pp. 2672–2680.
[89] M. Mathieu, C. Couprie, Y. LeCun, Deep multi-scale video prediction beyond mean square error, in: Proc.
International Conference on Learning Representations (ICLR), 2016.
[90] R. Girshick, J. Donahue, T. Darrell, J. Malik, Rich feature hierarchies for accurate object detection and semantic
segmentation, in: Computer Vision and Pattern Recognition, 2014.
[91] R. Girshick, Fast r-cnn, in: Proceedings of the IEEE International Conference on Computer Vision, 2015, pp.
1440–1448.
[92] S. Ren, K. He, R. Girshick, J. Sun, Faster r-cnn: Towards real-time object detection with region proposal
networks, in: Advances in neural information processing systems, 2015, pp. 91–99.
[93] G. Gkioxari, J. Malik, Finding action tubes, in: Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, 2015, pp. 759–768.
[94] P. Weinzaepfel, Z. Harchaoui, C. Schmid, Learning to track for spatio-temporal action localization, in: Proceedings of the IEEE International Conference on Computer Vision, 2015, pp. 3164–3172.
42

[95] Z. Shou, D. Wang, S.-F. Chang, Temporal action localization in untrimmed videos via multi-stage cnns, in:
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 1049–1058.
[96] X. Peng, C. Schmid, Multi-region two-stream r-cnn for action detection, in: European Conference on Computer
Vision, Springer, 2016, pp. 744–759.
[97] L. Wang, Y. Xiong, D. Lin, L. Van Gool, Untrimmednets for weakly supervised action recognition and detection,
in: CVPR, 2017.
[98] Y. Zhao, Y. Xiong, L. Wang, Z. Wu, D. Lin, X. Tang, Temporal action detection with structured segment
networks, in: ICCV, 2017.
[99] H. Zhu, R. Vial, S. Lu, Tornado: A spatio-temporal convolutional regression network for video action proposal,
in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 5813–5821.
[100] R. Hou, C. Chen, M. Shah, Tube convolutional neural network (t-cnn) for action detection in videos, in: ICCV,
2017.
[101] S. Saha, G. Singh, F. Cuzzolin, Amtnet: Action-micro-tube regression by end-to-end trainable deep architecture,
in: ICCV, 2017.
[102] V. Kalogeiton, P. Weinzaepfel, V. Ferrari, C. Schmid, Action tubelet detector for spatio-temporal action localization, in: ICCV, 2017.
[103] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, A. C. Berg, Ssd: Single shot multibox detector,
in: ECCV, Springer, 2016, pp. 21–37.
[104] J. Gao, Z. Yang, C. Sun, K. Chen, R. Nevatia, Turn tap: Temporal unit regression network for temporal action
proposals, in: ICCV, 2017.
[105] G. Singh, S. Saha, M. Sapienza, P. Torr, F. Cuzzolin, Online real-time multiple spatiotemporal action localisation and prediction, in: ICCV, 2017.
[106] C. Lea, M. D. Flynn, R. Vidal, A. Reiter, G. D. Hager, Temporal convolutional networks for action segmentation
and detection, in: CVPR, 2017.
[107] D.-A. Huang, L. Fei-Fei, J. C. Niebles, Connectionist temporal modeling for weakly supervised action labeling,
in: European Conference on Computer Vision, Springer, 2016, pp. 137–153.
[108] S. Buch, V. Escorcia, C. Shen, B. Ghanem, J. C. Niebles, Sst: Single-stream temporal action proposals, in:
CVPR, 2017.
[109] S. Yeung, O. Russakovsky, G. Mori, L. Fei-Fei, End-to-end learning of action detection from frame glimpses
in videos, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp.
2678–2687.
43

[110] B. Singh, T. K. Marks, M. Jones, O. Tuzel, M. Shao, A multi-stream bi-directional recurrent neural network
for fine-grained action detection, in: Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition, 2016, pp. 1961–1970.
[111] S. Ma, L. Sigal, S. Sclaroff, Learning activity progression in lstms for activity detection and early detection, in:
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 1942–1950.
[112] A. Dave, O. Russakovsky, D. Ramanan, Predictive-corrective networks for action detection, in: CVPR, 2017.
[113] Z. Liu, C. Zhang, Y. Tian, 3d-based deep convolutional neural network for action recognition with depth sequences, Image and Vision Computing 55 (2016) 93–100.
[114] P. Wang, W. Li, Z. Gao, C. Tang, J. Zhang, P. O. Ogunbona, Convnets-based action recognition from depth
maps through virtual cameras and pseudocoloring, in: ACM MM, 2015.
[115] P. Wang, W. Li, Z. Gao, J. Zhang, C. Tang, P. Ogunbona, Action recognition from depth maps using deep
convolutional neural networks, THMS 46 (4) (2016) 498–509.
[116] X. Yang, C. Zhang, Y. Tian, Recognizing actions using depth motion maps-based histograms of oriented gradients, in: ACM MM, 2012.
[117] H. Rahmani, A. Mian, 3d action recognition from novel viewpoints, in: Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition, 2016, pp. 1506–1515.
[118] P. Wang, W. Li, S. Liu, Y. Zhang, Z. Gao, P. Ogunbona, Large-scale continuous gesture recognition using
convolutional neural networks, in: 2016 23rd International Conference on Pattern Recognition (ICPR), 2016,
pp. 13–18.
[119] F. Jiang, S. Zhang, S. Wu, Y. Gao, D. Zhao, Multi-layered gesture recognition with kinect, Journal of Machine
Learning Research 16 (2) (2015) 227–254.
[120] Y. Du, Y. Fu, L. Wang, Skeleton based action recognition with convolutional neural network, in: Pattern Recognition (ACPR), 2015 3rd IAPR Asian Conference on, IEEE, 2015, pp. 579–583.
[121] Y. Hou, Z. Li, P. Wang, W. Li, Skeleton optical spectra based action recognition using convolutional neural
networks, in: Circuits and Systems for Video Technology, IEEE Transactions on, 2016, pp. 1–5.
[122] C. Li, Y. Hou, P. Wang, W. Li, Joint distance maps based action recognition with convolutional neural networks,
IEEE Signal Processing Letters 24 (5) (2017) 624–628.
[123] M. Liu, H. Liu, C. Chen, Enhanced skeleton visualization for view invariant human action recognition, Pattern
Recognition 68 (2017) 346–362.
[124] Q. Ke, S. An, M. Bennamoun, F. Sohel, F. Boussaid, Skeletonnet: Mining deep part features for 3d action
recognition, IEEE Signal Processing Letters.
44

[125] Q. Ke, M. Bennamoun, S. An, F. Sohel, F. Boussaid, A new representation of skeleton sequences for 3d action
recognition, in: CVPR, 2017.
[126] T. S. Kim, A. Reiter, Interpretable 3d human action analysis with temporal convolutional networks, in: CVPR,
2017.
[127] Y. Du, W. Wang, L. Wang, Hierarchical recurrent neural network for skeleton based action recognition, in:
CVPR, 2015.
[128] Y. Du, Y. Fu, L. Wang, Representation learning of temporal dynamics for skeleton-based action recognition,
IEEE Transactions on Image Processing 25 (7) (2016) 3010–3022.
[129] V. Veeriah, N. Zhuang, G.-J. Qi, Differential recurrent neural networks for action recognition, in: ICCV, 2015.
[130] W. Zhu, C. Lan, J. Xing, W. Zeng, Y. Li, L. Shen, X. Xie, Co-occurrence feature learning for skeleton based
action recognition using regularized deep LSTM networks, in: AAAI, 2016.
[131] J. Liu, A. Shahroudy, D. Xu, G. Wang, Spatio-temporal LSTM with trust gates for 3D human action recognition,
in: ECCV, 2016.
[132] S. Song, C. Lan, J. Xing, W. Zeng, J. Liu, An end-to-end spatio-temporal attention model for human action
recognition from skeleton data, in: Thirty-First AAAI Conference on Artificial Intelligence, 2017.
[133] S. Zhang, X. Liu, J. Xiao, On geometric features for skeleton-based action recognition using multilayer lstm
networks, in: WACV, 2017.
[134] I. Lee, D. Kim, S. Kang, S. Lee, Ensemble deep learning for skeleton-based action recognition using temporal
sliding lstm networks, in: ICCV, 2017, pp. 1012–1020.
[135] W. Li, L. Wen, M.-C. Chang, S. Nam Lim, S. Lyu, Adaptive rnn tree for large-scale human action recognition,
in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 1444–1452.
[136] P. Zhang, C. Lan, J. Xing, W. Zeng, J. Xue, N. Zheng, View adaptive recurrent neural networks for high
performance human action recognition from skeleton data, in: ICCV, 2017.
[137] Y. Li, C. Lan, J. Xing, W. Zeng, C. Yuan, J. Liu, Online human action detection using joint classificationregression recurrent neural networks, in: European Conference on Computer Vision, Springer, 2016, pp. 203–
220.
[138] Z. Huang, C. Wan, T. Probst, L. Van Gool, Deep learning on lie groups for skeleton-based action recognition,
in: CVPR, 2017.
[139] R. Salakhutdinov, J. B. Tenenbaum, A. Torralba, Learning with hierarchical-deep models, IEEE Transactions
on Pattern Analysis and Machine Intelligence 35 (8) (2013) 1958–1971.

45

[140] Y. W. Teh, M. I. Jordan, M. J. Beal, D. M. Blei, Sharing clusters among related groups: Hierarchical dirichlet
processes., in: NIPS, 2004, pp. 1385–1392.
[141] D. Wu, L. Shao, Leveraging hierarchical parametric networks for skeletal joints based action segmentation and
recognition, in: 2014 IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 724–731.
[142] E. P. Ijjina, et al., Classification of human actions using pose-based features and stacked auto encoder, Pattern
Recognition Letters 83 (2016) 268–277.
[143] G. Zhu, L. Zhang, P. Shen, J. Song, Multimodal gesture recognition using 3d convolution and convolutional
lstm, IEEE Access.
[144] G. Zhu, L. Zhang, L. Mei, J. Shao, J. Song, P. Shen, Large-scale isolated gesture recognition using pyramidal 3d
convolutional networks, in: Pattern Recognition (ICPR), 2016 23rd International Conference on, IEEE, 2016,
pp. 19–24.
[145] P. Wang, S. Wang, Z. Gao, Y. Hou, W. Li, Structured images for rgb-d action recognition, in: Proceedings of
the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 1005–1014.
[146] Y. Li, Q. Miao, K. Tian, Y. Fan, X. Xu, R. Li, J. Song, Large-scale gesture recognition with a fusion of rgb-d
data based on the c3d model, in: Pattern Recognition (ICPR), 2016 23rd International Conference on, IEEE,
2016, pp. 25–30.
[147] S. Xingjian, Z. Chen, H. Wang, D.-Y. Yeung, W.-K. Wong, W.-c. Woo, Convolutional lstm network: A machine
learning approach for precipitation nowcasting, in: Advances in Neural Information Processing Systems, 2015,
pp. 802–810.
[148] Z. Luo, B. Peng, D.-A. Huang, A. Alahi, L. Fei-Fei, Unsupervised learning of long-term motion dynamics for
videos, in: CVPR, 2017.
[149] Z. Shi, T.-K. Kim, Learning and refining of privileged information-based rnns for action recognition from depth
sequences, in: Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[150] B. Mahasseni, S. Todorovic, Regularizing long short term memory with 3d human-skeleton sequences for action
recognition, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp.
3054–3062.
[151] A. Shahroudy, T.-T. Ng, Y. Gong, G. Wang, Deep multimodal feature analysis for action recognition in rgb+ d
videos, IEEE Transactions on Pattern Analysis and Machine Intelligence.
[152] D. Wu, L. Pigou, P.-J. Kindermans, N. D.-H. Le, L. Shao, J. Dambre, J.-M. Odobez, Deep dynamic neural networks for multimodal gesture segmentation and recognition, IEEE transactions on pattern analysis and machine
intelligence 38 (8) (2016) 1583–1597.

46

[153] X. Chai, Z. Liu, F. Yin, Z. Liu, X. Chen, Two streams recurrent neural networks for large-scale continuous
gesture recognition, in: International Conference on Pattern Recognition Workshops, 2016.
[154] P. Molchanov, X. Yang, S. Gupta, K. Kim, S. Tyree, J. Kautz, Online detection and classification of dynamic
hand gestures with recurrent 3d convolutional neural network, in: Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition, 2016, pp. 4207–4215.
[155] A. Graves, S. Fernández, F. Gomez, J. Schmidhuber, Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks, in: Proceedings of the 23rd international conference on
Machine learning, ACM, 2006, pp. 369–376.
[156] N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, R. Salakhutdinov, Dropout: a simple way to prevent
neural networks from overfitting., Journal of Machine Learning Research 15 (1) (2014) 1929–1958.
[157] Y. Li, Q. Miao, K. Tian, Y. Fan, X. Xu, R. Li, J. Song., Large-scale gesture recognition with a fusion of RGB-D
data based on the C3D model, in: Proceedings of ICPRW, 2016.
[158] M. Zolfaghari, G. L. Oliveira, N. Sedaghat, T. Brox, Chained multi-stream networks exploiting pose, motion,
and appearance for action classification and detection, in: Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition, 2017.
[159] N. C. Camgoz, S. Hadfield, O. Koller, R. Bowden, Using convolutional 3d neural networks for user-independent
continuous gesture recognition, in: 2016 23rd International Conference on Pattern Recognition (ICPR), 2016,
pp. 49–54.
[160] L. Cao, Z. Liu, T. S. Huang, Cross-dataset action detection, in: Computer vision and pattern recognition
(CVPR), IEEE, 2010, pp. 1998–2005.
[161] W. Sultani, I. Saleemi, Human action recognition across datasets by foreground-weighted histogram decomposition, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014, pp. 764–771.
[162] J. Zhang, W. Li, P. Ogunbona, Joint geometrical and statistical alignment for visual domain adaptation, in:
CVPR, 2017.
[163] M. S. Ryoo, Human activity prediction: Early recognition of ongoing activities from streaming videos, in:
Computer Vision (ICCV), 2011 IEEE International Conference on, IEEE, 2011, pp. 1036–1043.
[164] M. Hoai, F. De la Torre, Max-margin early event detectors, International Journal of Computer Vision 107 (2)
(2014) 191–202.
[165] T. Lan, T.-C. Chen, S. Savarese, A hierarchical representation for future action prediction, in: European Conference on Computer Vision, Springer, 2014, pp. 689–704.

47

[166] T.-H. Vu, C. Olsson, I. Laptev, A. Oliva, J. Sivic, Predicting actions from static scenes, in: European Conference
on Computer Vision, Springer, 2014, pp. 421–436.
[167] M. Sadegh Aliakbarian, F. Sadat Saleh, M. Salzmann, B. Fernando, L. Petersson, L. Andersson, Encouraging
lstms to anticipate actions very early, in: ICCV, 2017, pp. 280–289.
[168] J. Wan, G. Guo, S. Z. Li, Explore efficient local features from RGB-D data for one-shot learning gesture
recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence 38 (8) (2016) 1626–1639.
[169] I. Guyon, V. Athitsos, P. Jangyodsuk, H. J. Escalante, The chalearn gesture dataset (CGD 2011), Machine Vision
and Applications 25 (8) (2014) 1929–1951.
[170] N. Madapana, J. P. Wachs, A semantical & analytical approach for zero shot gesture learning, in: FG workshop,
2017.
[171] J. Wan, Q. Ruan, W. Li, S. Deng, One-shot learning gesture recognition from RGB-D data using bag of features,
Journal of Machine Learning Research 14 (2013) 2549–2582.
[172] P. Mettes, C. G. Snoek, Spatial-aware object embeddings for zero-shot localization and classification of actions,
in: ICCV, 2017.
[173] J. Ho, S. Ermon, Generative adversarial imitation learning, in: Advances in Neural Information Processing
Systems, 2016, pp. 4565–4573.
[174] L. Tran, X. Yin, X. Liu, Disentangled representation learning gan for pose-invariant face recognition, in: CVPR,
Vol. 4, 2017, p. 7.
[175] C. Vondrick, H. Pirsiavash, A. Torralba, Generating videos with scene dynamics, in: Advances In Neural
Information Processing Systems, 2016, pp. 613–621.
[176] C. Vondrick, H. Pirsiavash, A. Torralba, Anticipating visual representations from unlabeled video, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 98–106.

48

