Learning to Recognize 3D Human Action from A New Skeleton-based Representation Using Deep Convolutional Neural Networks
Huy-Hieu Pham Louahdi Khoudour Alain Crouzil Pablo Zegers Sergio A. Velastin







1Cerema, Equipe-projet STI, 1 Avenue du Colonel Roche, 31400, Toulouse, France
2Institut de Recherche en Informatique de Toulouse (IRIT), Université de Toulouse, UPS, 31062 Toulouse, France
3Aparnix, La Gioconda 4355, 10B, Las Condes, Santiago, Chile
4Department of Computer Science, Applied Artificial Intelligence Research Group, University Carlos III de Madrid, 28270 Madrid, Spain
5School of Electronic Engineering and Computer Science, Queen Mary University of London, UK
6Cortexica Vision Systems Ltd., London, UK
 huy-hieu.pham@cerema.fr


Recognizing human actions in untrimmed videos is an important challenging task. An effective 3D motion representation and a powerful learning model are two key factors influencing recognition performance. In this paper we introduce a new skeleton-based representation for 3D action recognition in videos. The key idea of the proposed representation is to transform 3D joint coordinates of the human body carried in skeleton sequences into RGB images via a color encoding process. By normalizing the 3D joint coordinates and dividing each skeleton frame into five parts, where the joints are concatenated according to the order of their physical connections, the color-coded representation is able to represent spatio-temporal evolutions of complex 3D motions, independently of the length of each sequence. We then design and train different Deep Convolutional Neural Networks (D-CNNs) based on the Residual Network architecture (ResNet) on the obtained image-based representations to learn 3D motion features and classify them into classes. Our method is evaluated on two widely used action recognition benchmarks: MSR Action3D and NTU-RGB+D, a very large-scale dataset for 3D human action recognition. The experimental results demonstrate that the proposed method outperforms previous state-of-the-art approaches whilst requiring less computation for training and prediction.


Learning to Recognize 3D Human Action from A New Skeleton-based Representation Using Deep Convolutional Neural Networks
    Huy-Hieu Pham ^1,2,*, 
Louahdi Khoudour ^1, 
Alain Crouzil ^2, 
Pablo Zegers ^3, 
Sergio A. Velastin ^4,5,6

    Received: date / Accepted: date
=======================================================================================================================





§ INTRODUCTION


Human Action Recognition (HAR) <cit.> is an important topic in machine vision. HAR methods have been widely applied in building electronic systems based on machine intelligence such as intelligent surveillance <cit.>, human-machine interaction <cit.>, health
care <cit.>, and so on. Although significant progress has been made in the last few years, HAR is still a challenging task due to many obstacles, e.g. camera viewpoint, occlusions, or large intra-class variations <cit.>.

 There are two main types of intelligence that
are artificial intelligence and machine intelligence. HAR can be considered as an electronic system based on machine intelligence. The first stage of any HAR system is data acquisition. Nowadays, a variety of electronic imaging systems can be used for this task such as traditional optical cameras, RGB-D sensors, Thermal Infrared (IR) sensors, and Synthetic Aperture Radar (SAR) sensors. For instance, IR sensors have been exploited in HAR <cit.>. These sensors are able to generate images based on the heat radiated by the human body and to work independently of lighting conditions. SAR imaging has also been widely used for detecting and analyzing human activities <cit.>. This type of sensor is able to operate far away from potential targets and functions during the daytime as well as nighttime, under all weather conditions. In particular, SAR Image Segmentation (SIS) techniques <cit.> can segment humans from other components (e.g. objects, background) in the scene. As a result, the use of SAR sensors offers big advantages for human action recognition, in particular for human target detection and identification in a complex environment such as in military applications. 

 Traditional studies on HAR mainly focus on the use of RGB sequences provided by 2D cameras. These approaches typically recognize actions based on hand-crafted local features such as Cuboids <cit.>, HOG/HOF <cit.>, HOG-3D <cit.>, which are extracted from the appearance and movements of human body parts. However, it is very hard to fully capture and model the spatial-temporal information of human action due to the absence of 3D structure from the scene. Therefore, RGB-D cameras are becoming one of the most commonly-used sensors for HAR <cit.>. Recently, the rapid development of cost-effective and easy-to-use depth cameras, e.g. Microsoft Kinect ^TMsensor <cit.>, ASUS Xtion PRO <cit.>, or Intel^® RealSense^TM <cit.> has helped dealing with problems related to 3D action recognition. In general, depth sensors are able to provide detailed information about the 3D structure of human body. Therefore, many approaches have been proposed for 3D HAR based on data provided by depth sensors such as RGB sequences, depth, or combining these two data types (RGB-D). Furthermore, these devices have integrated real-time skeleton tracking algorithms <cit.> that provide high-level information about human motions in a 3D space. Thus, exploiting skeletal data for 3D HAR is an active research topic in computer vision <cit.>.

 In recent years, approaches based on Convolutional Neural Networks (CNNs) have achieved outstanding results in many image recognition tasks <cit.>. After the success of AlexNet <cit.>, a new direction of research has been opened for designing and optimizing higher performing CNN architectures. As a result, there are substantial evidences that seem to show that the learning performance of CNNs may be significantly improved by increasing the number of hidden layers <cit.>. In the literature of HAR, many studies have indicated that CNNs have the ability to learn complex motion features better than hand-crafted approaches. However, most previous works have just focused on exploring simple CNNs such as AlexNet <cit.> and have not exploited the potential of recent state-of-the-art very deep CNN architectures (D-CNNs), e.g. Residual Networks (ResNet) <cit.>. In addition, most existing CNN-based approaches limit themselves to using RGB-D sequences as the input to learning models. Although RGB-D images are informative for understanding human action, the computation complexity of learning models increases rapidly when the dimension of the input features is large. Therefore, representation learning models based on RGB-D modality become more complex, slower and less practical for solving large-scale problems as well as real-time applications.

 Different from previous works, to take full advantages of 3D skeletal data and the learning capacity of D-CNNs, this paper proposes an end-to-end deep learning framework for 3D HAR from skeleton sequences. We focus on solving two main issues: First, using a simple skeleton-to-image encoding method to transform the 3D coordinates of the skeletal joints into RGB images. The encoding method needs to ensure that the image-based representation of skeleton sequences is able to effectively represent the spatial structure and temporal dynamics of the human action. Second, we design and train different D-CNNs based on Residual Networks (ResNets) <cit.> – a recent state-of-the-art CNN for image recognition, to learn and classify actions from the obtained image-coded representations. 

 This paper is an extended version of our work published in the 8th International Conference of Pattern Recognition Systems (ICPRS 2017) <cit.>. It is part of our research project about investigating and developing a low-cost system based on RGB-D data provided by depth sensors for understanding human actions and analyzing their behaviors in indoor environments such as inside home, offices, residential buildings, buses, trains, etc. In general, three observations motivate our exploration of using ResNet <cit.> for 3D HAR from skeleton sequences, including: (1) human actions can be correctly
represented as the movements of skeletons <cit.> and the spatial-temporal dynamics of skeletons can be transformed into 2D image structure, which can be effectively learned by D-CNNs; (2) skeletal data is high-level information with much less complexity than RGB-D sequences, this advantage makes our action learning model much simpler and faster; (3) many evidences <cit.> show that deeper convolutional model, especially ResNet <cit.> can boost the learning accuracy in image recognition tasks.

 We evaluate the proposed method on two benchmark skeleton datasets (i.e. MSR Action3D <cit.> and NTU-RGB+D <cit.>). Experimental results confirmed the above statements since our method achieveds state-of-the-art performance compared with the existing results using the same evaluation protocols. Furthermore, we also indicate the effectiveness of our learning framework in terms of computational complexity.

 In summary, two main contributions of this paper include:



∙ First, we introduce a new skeleton-based representation (Sec. <ref>) and an end-to-end learning framework[Codes and pre-trained models will be shared to the community at <https://github.com/huyhieupham/> after publication.] (Sec. <ref>)  based on D-CNNs to learn the spatio-temporal evolutions of 3D motions and then to recognize human actions;



∙ Second, we show the effectiveness of our method on HAR tasks by achieving state-of-the-art performance on two benchmark datasets, including the most challenging skeleton benchmark currently available for 3D HAR <cit.> (Sec. <ref>).



The advantage of the proposed method is that it has high computational efficiency (Sec. <ref>). Moreover, this approach is general and can be easily applied to other time-series problem, e.g. recognizing human actions with mobile devices with integrated inertial sensors. 

 The rest of the paper is organized as follows: Section <ref> discusses related works. In Section <ref>, we present the details of our proposed method, including the color-encoding process from skeletons to RGB images and deep learning networks. Datasets and experiments are described in Section <ref>. Experimental results are reported in Section <ref> with a detailed analysis of computational efficiency. Finally, Section <ref> concludes the paper with a discussion on the future work.


§ RELATED WORK


Skeleton-based Action Recognition (SBAR) using depth sensors has been widely studied in recent years. In this section, we briefly review existing SBAR methods, including two main categories that are directly related to our work. The first category is approaches based on hand-crafted local features of skeletons. The second is approaches based on deep learning networks, especially Recurrent Neural Networks with Long Short Term Memory (RNN-LSTMs).



Approaches based on hand-crafted local features: Human motion can be considered as a spatio-temporal pattern. Many researchers have built hand-crafted feature representations for SBAR and then used temporal models for modeling 3D human motion. For instance, Wang et al. <cit.> represented the human motion from skeletal data by means of the pairwise relative positions of the key joints. The authors then employed Hidden Markov Model (HMM) <cit.> for modeling temporal dynamics of actions. A similar approach has been presented by Lv et al. <cit.>  where the 3D joint position trajectories have been mapped into feature spaces. The dynamics of actions was then learned by one continuous HMM. Xia et al. <cit.> proposed the use of histogram-based representation of human pose, and then actions were classified into classes by a discrete HMM. Wu and Shao <cit.> also exploited HMM for recognizing actions from high-level features of skeletons. Vemulapalli et al. <cit.> represented the 3D geometric interactions of the body parts in a Lie Group. These elements were processed with a Fourier Temporal Pyramid (FTP) transformation for modeling temporal evolutions of the original motions. Different from the studies above, Luo et al. <cit.> proposed a new dictionary learning model to learn the spatio-temporal information from skeleton sequences. Although promising results have been achieved from approaches based on hand-crafted local features and probabilistic graphical models, they have some limitations that are very difficult to overcome. E.g., most of these
approaches are data-dependent and require a lot of hand-designing features. HMM-based methods require preprocessing input data in which the skeleton sequences need to be segmented or aligned. Meanwhile, FTP-based approaches can only utilize limited contextual information of actions.



Approaches based on deep learning: Recurrent Neural Networks with Long Short-Term Memory Network (RNN-LSTMs) <cit.> are able to model the contextual information of the temporal sequences as skeleton data. Thus, many authors have explored RNN-LSTMs for SAR. For instance, Du et al. <cit.> proposed an end-to-end hierarchical RNN-LSTM for modeling local motions of a body part in which all skeleton frames were divided into five parts according to the human physical structure. Each part of a skeleton  was then fed into an independent  RNN and then fused to be the inputs of higher layers. The final representation was used for the classification task. Zhu et al. <cit.> proposed a LSTM network with a mixed-norm regularization term to cost functions in order to learn the co-occurrence of discriminative features from skeletal data. Liu et al. <cit.> introduced a new gating mechanism with LSTM network to analyze the reliability of input skeleton sequences. In another study, Shahroudy et al. <cit.> used a part-aware LSTM network in which the memory-cell was divided into sub-cells such that each sub-cell can model long-term contextual representation of a body part. Finally, all these sub-cells were concatenated to the final output. Although RNN-LSTMs are able to model the long-term temporal of motion and experimental results provided that RNN-LSTMs have outperformed many other approaches, RNN-based approaches just consider skeleton sequences as a kind of low-level feature by feeding directly the raw 3D joint coordinates carried in skeletons into the network input. The huge number of input parameters may make RNNs become very complex and easily lead to overfitting due to insufficient training data. Moreover, many RNN-LSTMs act as a classifier and cannot extract high-level features <cit.> for recognition tasks.

 Several authors have exploited the feature learning ability of CNNs on skeletal data <cit.>. However, such studies mainly focus on the use of complex encoding methods for finding good skeletal representations and learning geometric features carried in skeleton sequences with simple CNN architectures. In contrast, in this paper we concentrate on proposing a new and simple skeleton-based representation and  exploiting the power of D-CNNs for action recognition. Our experiments on two public datasets, including the MSR Action3D dataset <cit.> and the NTU-RGB+D dataset <cit.> show state-of-the-art performance on both datasets. Moreover, in terms of computational cost, our measurement and analyses show that the proposed model is fast enough for many real-time applications.



§ PROPOSED METHOD


This section presents the proposed method. We first describe the color-encoding process that allows us to represent the spatial structure and temporal dynamics of skeleton sequences as the static structure of a 2D color image. We then review the main idea of ResNet <cit.> and propose five different ResNet configurations, including 20-layer, 32-layer, 44-layer, 56-layer, and 110-layer networks to learn motion features carried in the image-based representations and perform action classification.


 §.§ Building image-based representation from skeleton sequences
 

Currently, skeletal data that contains 3D joint coordinates can be obtained from depth cameras via real-time skeleton estimation algorithms <cit.>. This technology allows to extract the position of the key joints in the body, which is suitable for 3D HAR problems. E.g., the latest version of the Microsoft Kinect ^TMsensor <cit.> (Kinect v2 sensor) can track the main 25 joints of human body in real-time speed as shown in Figure <ref>.


 In this paper, instead of feeding raw 3D joint coordinates directly to RNN-LSTMs as in many previous works <cit.>, we propose the use of D-CNNs for learning motion features from skeleton sequences. The first step is to find a skeleton-based representation that can effectively capture the spatio-temporal evolutions of skeletons and that can also be easily learned by learning method as D-CNNs. One solution for this problem is to transform each skeleton sequence into a single color image (e.g., RGB image) in which the pixel values must have the ability to represent the movement of skeletons.

 To this end, we transform the 3D joint coordinates of each skeleton into a new space by normalizing their coordinates using a transformation function. Specifically, given a skeleton sequence 𝒮 with N frames [F_1, F_2,..., F_N], let (x_i,y_i,z_i) be the 3D coordinates of each joint in frame {F_n}∈𝒮, n ∈ [1, N]. A normalization function ℱ(·) is used to transform all 3D joint coordinates to the range of [0, 255] as follows:

    (x'_i,y'_i,z'_i) = ℱ(x_i,y_i,z_i)
 

    x'_i = 255 ×(x_i - min{𝒞})max{𝒞} - min{𝒞}
 

    y'_i = 255 ×(y_i - min{𝒞})max{𝒞} - min{𝒞}


    z'_i = 255 ×(z_i - min{𝒞})max{𝒞} - min{𝒞}
 
where (x'_i,y'_i,z'_i) is 3D joint coordinate in the normalized space 𝒮^'. The max{𝒞} and min{𝒞} are the maximum and minimum values of all coordinates, respectively. To preserve the spatio-temporal information of skeleton movements, we stack all normalized frames according to the temporal order 𝒮^' = [F_1^',F_2^',...,F_N^'] to represent the whole action sequence. These elements are quantified to RGB color space and can be stored as RGB images. In this way, we convert the skeletal data to 3D tensors that will can then be fed into deep learning networks as the input for feature leaning and classification. Figure <ref> illustrates the skeleton-to-image transformation process. 



 Naturally, the human body is structured as four limbs and one trunk (see Figure <ref>). Simple actions can be performed through the movement of a limb, e.g., hand-waving, kicking forward, etc. More complex actions combine the movements of a group of limbs or the whole body, e.g., running or swimming. To keep the local characteristics of the human action <cit.>, we divide each skeleton into five parts, including two arms (P_1, P_2), two legs (P_4, P_5), and one trunk (P_3). In each part, the joints are concatenated according their physical connections as shown in Figure <ref>. These parts are then connected in a sequential order:  P_1 →P_2 →P_3 →P_4 →P_5. The whole process of rearranging all frames in a sequence can be done by rearranging the order of rows of pixels in the color-based representation. This process is illustrated in Figure <ref>.




Like that, we have encoded skeleton sequences into RGB images. Figure <ref> shows some examples of RGB images obtained from input sequences of MSR Action3D dataset <cit.>. These images will be learned and classified by D-CNN models. This way, the original skeleton sequence will be recognized via the corresponding image.  




 §.§ Deep residual networks for skeleton-based human action recognition
 

In this section, we introduce our network designs for recognizing human action from image-based representation obtained in Section <ref>. Before that, in order to put our method into context, we will briefly review the key idea of ResNet <cit.>.


  §.§.§ Residual learning method

Very deep neural networks, especially D-CNNs have demonstrated to have a high performance on many visual-related tasks <cit.>. However, D-CNNs are very difficult to optimize due to the vanishing gradients problem. If the network is deep enough, the error signal from the output layer can be completely attenuated on its way back to the input layer. In addition, another problem called “the degradation phenomenon <cit.>” also impedes the convergence of deeper networks. More specially, adding more layers to a deep network can lead to higher training/testing error. To overcome these challenges, ResNet <cit.> has been introduced. The key idea behind ResNet architectures is the presence of shortcut connections between input and output of each ResNet building block. These shortcut connections are implemented by identity mappings and are able to provide a path for gradients to back propagate to early layers in the network. This idea improves the information flow in ResNets and helps them to learn faster. Not only that, experimental results on standard datasets such as CIFAR-10 <cit.>, and ImageNet <cit.> for image classification tasks showed that the use of the shortcut connections in ResNet architectures can boost recognition performance. Figure <ref> shows information flow in a ResNet building block and their implementation details.

 Mathematically, a layer or a series of layers in a typical CNN model will try to learn a mapping function  y = ℱ(x) from input feature x. However, a ResNet building block will learn the function y = ℱ(x) + id(x) where id(x) is a identity function id(x) = x. The additional element id(x) is the key factor that improves the information flow through layers during training ResNets. 
 


  §.§.§ Network design

A deep ResNet can be constructed from multiple ResNet building blocks that are serially connected to each other. To search for the best recognition performance, we suggest different configurations of ResNet with 20, 32, 44, 56, and 110 layers. For further details, see Appendix. These networks are denoted as ResNet-20, ResNet-32, ResNet-44, ResNet-56, and ResNet-110, respectively. In our implementations, a ResNet building block performs the learning of a function y = ℱ(x) + id(x) where id(x) = x and ℱ(x) is implemented by a sequence of layers: Conv-BN-ReLU-Dropout-Conv-BN. Specifically, each ResNet block uses the convolutional layers (Conv) with 3×3 filters. The Batch Normalization (BN) <cit.> and non-linear activation function ReLU <cit.> are applied after each Conv. To reduce possible overfitting, we add a Dropout layer <cit.> with a rate of 0.5 into each ResNet block, located between two Conv layers and after ReLU. Finally, another ReLU layer <cit.> is used after the element-wise addition. The proposed ResNet unit is shown in Figure <ref>. We refer the interested reader to the Supplementary Materials to see details of the proposed network architectures. 


 In this learning framework, all networks are designed for accepting images with size 32×32 pixels and trained in an end-to-end manner using Stochastic Gradient Descent (SGD) algorithm <cit.> from scratch. The last fully-connected layer of the network represents the action class scores and its size can be changed corresponding to the number of action classes.


  §.§.§ Learning to recognize actions from skeleton-based representations

In order to recognize an action from a given skeleton sequence 𝒮 with N frames [F_1, F_2,..., F_N],  we firstly encode all frames of 𝒮 into a single RGB image I_RGB as mentioned in section <ref> 

    I_RGB = [F_1^',F_2^',...,F_N^'],

where all elements of F_i | i ∈ [1, N]^' were normalized to the range of [0, 255] as color pixels and rearranged according to their physical structure (see Figure <ref>). We then propose the use of ResNets to learn and classify the obtained color images. During the training phase, we minimize the cross-entropy loss function ℒ_𝒳(y,ŷ) between the action labels y of I_RGB and the predicted action labels ŷ by the network over the training set 𝒳. In other words, the network will be trained to solve the following problem

    Arg𝒲min(ℒ_𝒳(y,ŷ)) = Arg𝒲min (   - 1/M∑_i = 1^M∑_j = 1^Cy_ijlogŷ_ij ),

where 𝒲 is the set of weights that will be learned by the network, M denotes the number of samples in training set 𝒳 and C is the number of action classes.


§ EXPERIMENTS



In this section, we evaluate the effectiveness of the proposed method on two benchmark datasets: MSR Action3D <cit.> and NTU-RGB+D <cit.>. To have a fair comparison with state-of-the-art approaches in the literature, we follow the evaluation protocols as provided in the original papers <cit.>. Recognition performance will be measured by average classification accuracy.


 §.§ Datasets and experimental protocols




  §.§.§ MSR Action3D Dataset <cit.>

This dataset[The MSR Action3D dataset can be obtained at: <https://www.uow.edu.au/w̃anqing/#Datasets>] was collected with the Microsoft Kinect v1 ^TMsensor. It contains 20 different actions. Each action was performed by 10 subjects for two or three times. The experiments were conducted on 557 valid sequences of the dataset after removing the defective sequences. We followed the same experimental protocol as many other authors, in which the whole data is divided into three subsets named as AS1, AS2, and AS3. For each subset, five subjects (with IDs: 1, 3, 5, 7, 9) are selected for training and the rest (with IDs: 2, 4, 6, 8, 10) for test. More details about the action classes in each subset can be found in Table <ref>.



  §.§.§ NTU-RGB+D Dataset <cit.>


The NTU-RGB+D dataset[Obtained at: <http://rose1.ntu.edu.sg/Datasets/actionRecognition.asp>] is a very large-scale RGB+D dataset. To the best of our knowledge, the NTU-RGB+D is currently the largest dataset that contains skeletal data for HAR. It is a very challenging dataset due to the large intra-class variations and multiple viewpoints. Specifically, the NTU-RGB+D provides more than 56,000 videos, collected from 40 subjects for 60 action classes. The list of action classes in NTU RGB+D dataset includes: drinking, eating, brushing teeth, brushing hair, dropping, picking up, throwing, sitting down, standing up (from sitting position), clapping, reading, writing, tearing up paper, wearing jacket, taking off jacket, wearing a shoe, taking off a shoe, wearing on glasses, taking off glasses, putting on a hat/cap, taking off a hat/cap, cheering up, hand waving, kicking something, reaching into self pocket, hopping, jumping up, making/answering a phone call, playing with phone, typing, pointing to something, taking selfie, checking time (on watch), rubbing two hands together, bowing, shaking head, wiping face, saluting, putting palms together, crossing hands in front, sneezing/coughing, staggering, falling down, touching head (headache), touching chest (stomachache/heart pain), touching back (back-pain), touching neck (neck-ache), vomiting, fanning self, punching/slapping other person, kicking other person, pushing other person, patting other's back, pointing to the other person, hugging, giving something to other person, touching other person's pocket, handshaking, walking towards each other, and walking apart from each other. 

 The NTU RGB+D data was collected by the Microsoft Kinect v2 ^TMsensor. Each skeleton frame provides the 3D coordinates of 25 body joints. Two different evaluation criteria have been suggested, including Cross-Subject and Cross-View. In the Cross-Subject settings, 20 subjects (with IDs: 1, 2, 4, 5, 8, 9, 13, 14, 15, 16, 17, 18, 19, 25, 27, 28, 31, 34, 35, and 38) are used for training and the rest are used for test. For Cross-View settings, the sequences provided by cameras 2 and 3 are used for training and sequences provided by camera 1 are used for test.



 §.§ Data augmentation
  Very deep neural networks as ResNets <cit.> require a lot of data to optimize. D-CNNs cannot learn and classify images effectively when training with an insufficient data amount. Unfortunately, we have only 557 skeleton sequences (less than 30 images per class) on MSR Action3D dataset <cit.>. Therefore, data augmentation has been applied to prevent ResNets from overfitting. We used random cropping, flip horizontally and vertically techniques to generate more training samples. More specifically, 8 × cropping has been applied on 40  × 40 images to create 32 × 32 images. Then, their horizontally and vertically flipped images are also created. Figure <ref> illustrates some data augmentation techniques on an image sample obtained from the MSR Action3D dataset <cit.>. For the NTU-RGB+D dataset <cit.>, we do not apply any data augmentation method due to its very large-scale. 



 §.§ Implementation details
 
The image-based representations are computed directly from the original skeletons without any data preprocessing technique, e.g., transforming the coordinate system, using a fixed number of frames, or removing noise. In this project, the MatConvNet toolbox <cit.> was used to design and train our deep learning networks. During the training phase, we use mini-batches of 128 images for ResNet-20, ResNet-32, ResNet-44, and ResNet-56 networks. For ResNet-110 network, we use mini-batches of 64 images. We initialize the weights randomly and train all networks for 120 epochs on a computer using Geforce GTX 1080 Ti GPU with 11GB RAM from scratch. The initial learning rate is set to 0.01 and is decreased to 0.001 at epoch 75. The last 45 epochs use a learning rate of 0.0001. The weight decay is set at 0.0001 and the momentum is 0.9.



§ EXPERIMENTAL RESULTS AND ANALYSIS


This section reports our experimental results on the MSR Action3D dataset <cit.> and the NTU-RGB+D dataset <cit.>. We compare the obtained classification rates with state-of-the-art approaches in the literature <cit.>. In addition, we also provide a detailed analysis about the computational efficiency of the proposed learning framework.


 §.§ Result on MSR Action3D dataset

Experimental results on MSR Action3D dataset <cit.> are shown in Table <ref>. We achieved the best classification accuracy with the ResNet-32 model. More specifically, classification accuracies are 99.80% on AS1, 99.00% on AS2, and 99.80% on AS3. We obtained a total average accuracy of 99.53%. Our result outperforms many previous works <cit.> (see Table <ref>). Moreover, this result also indicates that the proposed ResNet architecture learns image features better than the original ResNet architecture <cit.>.   The learning curves of all proposed ResNets on AS1, AS2, AS3  are shown in Figure <ref>a, Figure <ref>b, and Figure <ref>c, respectively.




 §.§ Result on NTU-RGB+D dataset

As shown in Table <ref>, the proposed learning framework reaches competitive results with an accuracy of 73.40% on Cross-Subject evaluation and an accuracy of 80.40% on Cross-View evaluation. The obtained results indicate that our method can deal with large intra-class variations and multiple viewpoints dataset as NTU-RGB+D <cit.>. Table <ref> provides a comparison with published studies <cit.>. It is clear that our method surpasses many previous approaches in the same experimental conditions. The learning curves of all ResNet configurations on two evaluation settings are shown in Figure <ref>d and Figure <ref>e. 






 §.§ Convergence rate analysis

Figure <ref> shows the learning curves of the five proposed deep learning networks on the MSR Action3D <cit.> and the NTU-RGB+D <cit.> datasets. We can find that the training convergence rate of the networks are different on the two datasets. More specifically, on MSR Action3D dataset <cit.>, the proposed networks exhibit rapid convergence after 50 epochs at the beginning of the training process. Meanwhile, these networks can only start to converge after near 80 epochs on the NTU-RGB+D dataset <cit.>. This phenomenon can be explained by the complexity of feature spaces generated by the two datasets. With more than 56,000 videos collected from 40 subjects for 60 action classes, it is clear that the NTU-RGB+D dataset <cit.> is more complex than the MSR Action3D dataset <cit.>. This leads to deep learning networks needing more iterations to start convergence.

As shown in Table 2 and Table 4, the best recognition accuracy and configuration are highlighted in bold. More specifically, our experimental result on the MSR Action3D indicated that the baseline ResNet-32 achieved the best overall accuracy (99.53%). On the NTU RGB+D dataset, the ResNet-32 network was the best version on the Cross-View setting (80.40%). However, it worked worse than the ResNet-20 on the Cross-Subject setting. This is a weak aspect 
of the proposed approach. This limitation could be overcome by using ensemble learning techniques <cit.>.



 §.§ Analysis of training and prediction time
 
We take the AS1 subset of the MSR Action3D dataset <cit.> and the proposed ResNet-32 network for illustrating the computational efficiency of our deep learning framework. As shown in Figure <ref>, the proposed framework contains three main stages, including the encoding process from skeleton sequences into RGB images (Stage A),  the supervised training stage (Stage B), and the prediction stage (Stage C). With the implementation in Matlab using MatconvNet toolbox <cit.>[MatconvNet is an open source library and can be downloaded at address: <http://www.vlfeat.org/matconvnet/>.] on a single NVIDIA GeForce GTX 1080 Ti GPU[More details about the GPU specification, please refer to: <https://www.nvidia.com/en-us/geforce/products/10series/geforce-gtx-1080-ti/>.], without parallel processing, we take on average 4.15×10^-3s per sequence during training stage. While the prediction time, including the time for encoding skeletons into RGB images and classification by the pre-trained ResNet, takes on average 21.84×10^-3s per sequence (see Table <ref>). These results verify the effectiveness of our proposed method, not only in terms of accuracy but also in terms of computational efficiency.





§ CONCLUSION AND FUTURE WORK


In this paper, we have proposed a new 3D motion representation and an end-to-end deep learning framework based on ResNets <cit.> for human action recognition from skeleton sequences. To this end, we transformed the 3D joint coordinates carried in skeleton sequences into RGB images via a color encoding process. We then designed and trained different D-CNNs based on the ResNet architecture for learning the spatial and temporal dynamics of human motion from image-coded representations. The experimental results on two well-established datasets, including the largest RGB-D dataset currently available, demonstrated that our method can achieve state-of-the-art performance whilst requiring a low computation cost for the training and prediction stages.
For future work, we aim to extend this research by investigating a new skeleton encoding method, in which the Euclidean distances and orientation relations between joints will be exploited. In addition, we plan to build a better feature learning and classification framework by experimenting some new and potential architectures based on the idea of ResNet such as Inception Residual Networks (Inception-ResNets) <cit.> or Densely Connected Convolutional Networks (DenseNet <cit.>). In order to overcome the limitations of Kinect sensors when dealing with HAR in which the lighting conditions are available, we will also consider some new approaches for estimating human body key-points such as deep learning based approaches <cit.>, KWE and KCE methods <cit.>. We hope that our study will open a new door for the computer vision community on exploiting the potentials of very deep convolutional neural networks and skeletal data for 3D human action recognition.




§ ACKNOWLEDGMENT


This research was carried out at the Cerema Research Center (CEREMA) and Toulouse Institute of Computer Science Research (IRIT), Toulouse, France. Sergio A. Velastin is grateful for funding received from the Universidad Carlos III de Madrid, the European Union's Seventh Framework Programme for Research, Technological Development and demonstration under grant agreement N. 600371, el Ministerio de Economia, Industria y Competitividad (COFUND2013-51509) el Ministerio de Educación, cultura y Deporte (CEI-15-17) and Banco Santander.

ietConference



Appendix



This section describes the network architectures in detail. To build 20-layer, 32-layer, 44-layer, 56-layer, and 110-layer networks, we stack the proposed ResNet building units as following:




 Baseline 20-layer ResNet architecture 

3x3 Conv., 16 filters, BN, ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,16 filters-+-ReLU 

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,16 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,16 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,32 filters-+-ReLU



Residual unit: Conv.-BN-ReLU-Dropout-Conv.,32 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,32 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,64 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,64 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,64 filters-+-ReLU

Global mean pooling

FC layer with n units where n is equal the number of action class.

Softmax layer




 Baseline 32-layer ResNet architecture 

3x3 Conv., 16 filters, BN, ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,16 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,16 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,16 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,16 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,16 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,32 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,32 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,32 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,32 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,32 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,64 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,64 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,64 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,64 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,64 filters-+-ReLU

Global mean pooling

FC layer with n units where n is equal the number of action class.

Softmax layer






 Baseline 44-layer ResNet architecture 

3x3 Conv., 16 filters, BN, ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,16 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,16 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,16 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,16 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,16 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,16 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,16 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,32 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,32 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,32 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,32 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,32 filters-+-ReLU



Residual unit: Conv.-BN-ReLU-Dropout-Conv.,32 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,32 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,64 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,64 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,64 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,64 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,64 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,64 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,64 filters-+-ReLU

Global mean pooling

FC layer with n units, where n is equal the number of action class.

Softmax layer






 Baseline 56-layer ResNet architecture 

3x3 Conv., 16 filters, BN, ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,16 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,16 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,16 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,16 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,16 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,16 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,16 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,16 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,16 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,32 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,32 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,32 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,32 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,32 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,32 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,32 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,32 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,32 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,64 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,64 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,64 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,64 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,64 filters-+-ReLU



Residual unit: Conv.-BN-ReLU-Dropout-Conv.,64 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,64 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,64 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,64 filters-+-ReLU

Global mean pooling

FC layer with n units, where n is equal the number of action class.

Softmax layer






 Baseline 110-layer ResNet architecture 

3x3 Conv., 16 filters, BN, ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,16 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,16 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,16 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,16 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,16 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,16 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,16 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,16 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,16 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,16 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,16 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,16 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,16 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,16 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,16 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,16 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,16 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,16 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,32 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,32 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,32 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,32 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,32 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,32 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,32 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,32 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,32 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,32 filters-+-ReLU



Residual unit: Conv.-BN-ReLU-Dropout-Conv.,32 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,32 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,32 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,32 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,32 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,32 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,32 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,32 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,64 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,64 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,64 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,64 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,64 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,64 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,64 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,64 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,64 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,64 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,64 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,64 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,64 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,64 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,64 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,64 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,64 filters-+-ReLU

Residual unit: Conv.-BN-ReLU-Dropout-Conv.,64 filters-+-ReLU

Global mean pooling

FC layer with n units, where n is equal the number of action class.

Softmax layer


