Spatio-Temporal Graph Convolution for Skeleton Based Action Recognition
Chaolong Li Zhen Cui Wenming Zheng Chunyan Xu Jian Yang







Spatio-Temporal Graph Convolution for
 Skeleton Based Action Recognition
    Chaolong Li,^* Zhen Cui,Chaolong Li and Zhen Cui have equal contributions. Wenming Zheng, Chunyan Xu, Jian Yang 

School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, China

School of Biological Science & Medical Engineering, Southeast University, Nanjing, China

{zhen.cui, cyx, csjyang}@njust.edu.cn; {lichaolong, wenming_zheng}@seu.edu.cn
    December 30, 2023
=======================================================================================================================================================================================================================================================================================================================================================================================================


Variations of human body skeletons may be considered as dynamic graphs, which are generic data representation for numerous real-world applications. In this paper, we propose a spatio-temporal graph convolution (STGC) approach for assembling the successes of local convolutional filtering and sequence learning ability of autoregressive moving average. To encode dynamic graphs, the constructed multi-scale local graph convolution filters, consisting of matrices of local receptive fields and signal mappings, are recursively performed on structured graph data of temporal and spatial domain. The proposed model is generic and principled as it can be generalized into other dynamic models. We theoretically prove the stability of STGC and provide an upper-bound of the signal transformation to be learnt. Further, the proposed recursive model can be stacked into a multi-layer architecture. To evaluate our model, we conduct extensive experiments on four benchmark skeleton-based action datasets, including the large-scale challenging NTU RGB+D. The experimental results demonstrate the effectiveness of our proposed model and the improvement over the state-of-the-art.





§ INTRODUCTION


Human action recognition is one of the most active research topics due to its wide applications to video surveillance, robot vision, human computer interaction, etc. Since human body itself can be viewed as an articulated system of rigid bones connected by hinged joints, the actions of human body are essentially embodied in skeletal motions in the 3D space <cit.>. Thereby, various skeleton based action recognition methods <cit.> are springing up in recent years, accompanying with the progress of more accessible deep sensors.

Different from grid-shaped structures of images/videos, human skeleton, consisting of a series of joints and bones, has an irregular geometric structure. Human action may be regarded as a consecutive dynamic sequence of such irregular structures. Considering the motion complexity of the entire body skeleton, the sophisticated strategy is to separately model the trajectory of each joint or clique of joints (body part). The statistic model <cit.>, shape model <cit.>, and geometric model <cit.> are often used to characterize the motion trajectories. With the thriving of representation learning, the learning based methods, such as Hidden Markov Models (HMMs) <cit.> and recursive models (long-short term memory, LSTM) <cit.>, become predominant in the dynamic representation of skeletal joints, because they dedicate more promising results on those public skeleton datasets.

Recently graph as a tool is used to represent skeletons <cit.>, although graph is popular to model various structured objects. Nevertheless, the graph based method <cit.> still takes the conventional technique line of graph kernel matching.
In fact, in the research fields of graph classification/matching, the graph representation (, the statistic on graphlet <cit.>) and graph metric (, graph kernel <cit.>) have been historically well-studied. With recent successes of deep learning on various problems, deep representation of graphs has aroused more attention <cit.>. But the most crucial problem is the definition/identification of homogeneous graphs because the same responses should be produced from those homogeneous graphs. To this end, Niepert  <cit.> used a greedy strategy of sorting those nodes within a local neighbor region and then performed convolution-like filtering on the sorted nodes. Instead of this explicit spatial definition way, Defferrard  <cit.> introduced a deep graph method based on spectral filtering, inspired by the recent signal processing theory on graph <cit.>. Li  <cit.> introduced attention mechanism into graph convolution model. But those methods intrinsically belong to still-graph deep learning.

For dynamic graphs, some variants of recurrent neural network (RNN) are developed recently based on the traverse way of spatio-temporal graph. Jain  <cit.> proposed a structural-RNN by casting spatio-temporal graph as a RNN mixture for the task of action prediction. Li  <cit.> proposed gated graph sequential neural network for the basic logical reasoning task. Seo  <cit.> fed the spatial filtered graph signals into LSTM for image generation. However, these methods do not yet absorb the essential successes of convolutional neural networks (CNN), which have changed AI landscape with breakthrough results on numerous applications. 

In this paper, we propose a spatio-temporal graph convolution (STGC) approach to represent dynamic skeletal graph sequences. To encode the graph structure data, we design multi-scale convolutional filters, each of which is composed of receptive field computation and signal mapping. The receptive field is computed from the polynomial of adjacency matrix, which makes sure the same responses for homogeneous graphs. We simultaneously perform local convolutional filtering on temporal motions and spatial structures. The temporal convolutional filtering recursively encodes motion variations while the spatial filtering extracts more robust feature of spatial structures.
In theory, the frequency responses of multi-scale convolutional filtering are equivalent to an approximate graph Flourier transform following by one linear function of feature mapping. Taking the philosophy of multi-scale convolutional filtering, we develop a recursive graph convolution model inspired by autoregressive moving average (ARMA). We theoretically analyze the stability of the proposed model and provide a theoretical upper-bound. Finally, we stack the spatio-temporal graph convolution into a deep architecture. To verify our proposed method, we conduct extensive experiments on four benchmark skeleton-based action datasets including the largest NTU RGB+D dataset <cit.>. The experimental results demonstrate the effectiveness of our proposed model and a more promising direction for skeleton based action recognition.

In summary, our contributions are three folds:

  
  * Propose a spatio-temporal graph convolution approach, which assembles the successes of local convolutional filtering and sequence learning ability of recursive learning. The generic model is further extended to a deep architecture.
  
  * Theoretically prove the stability of the proposed model and provide a theoretical upper-bound.
  
  * Achieve the state-of-the-art performances on the four benchmark datasets including the large-scale challenging dataset NTU RGB+D.




§ GRAPH PRELIMINARY


Consider an undirected graph =(,) of N nodes, where ={v_i}_i=1^N is the set of nodes,  is a (weighted) adjacency matrix. The adjacency matrix ∈^N× N records the connections between nodes, where if v_i, v_j are not connected, then A_ij=0, otherwise A_ij≠ 0.

The graph Laplacian matrix Ł is defined as Ł = -, where ∈^N× N is the diagonal degree matrix with D_ii=∑_jA_ij. A popular option is the normalized version, , each weight A_ij is multiplied by a factor 1/√(D_iiD_jj), formally, Ł^norm = ^-1/2Ł^-1/2= -^1/2^1/2,
where  is the identity matrix. Unless otherwise specified, we use the normalized Laplacian matrix below.

As a symmetric semi-positive definite (SPD) matrix, the graph Laplacian Ł has a complete set of orthonormal eigenvectors {_1,⋯,_N} satisfying Ł_i = λ_i_i, where {λ_i} are nonnegative real eigenvalues. We assume all eigenvalues are ordered as 0=λ_1≤λ_2⋯≤λ_N=λ_max. For the normalized Laplacian matrix, we have a bound of λ_max=2. In matrix expression, the Laplacian matrix can be written as Ł=Λ, where Λ=([λ_1,λ_2,⋯,λ_N]) and =[_1;_2;⋯;_N].

According to graph theory <cit.>, the graph Fourier transform (GFT) of a signal  in spatial domain can be defined as =, where  is the produced graph frequency signal. The corresponding inverse GFT is =. A graph filtering $̋ is an operator that acts upon a graph signalby amplifying or attenuating its graph Fourier coefficients as= ∑_n=1^N H(λ_n)_n _n. The graph frequency responseH:[λ_min,λ_max]→controls how much$̋ amplifies the signal spectra H(λ_n)=(_n)/_n.



§ THE MODEL






Human body skeleton is represented with a group of 3D spatial coordinates of joints. Hence, one can use a graph to depict spatial relation of skeletal joints. To keep the original coordinate information, we add attributes of nodes into the graph, , =(,,), where ={v_i}_i=1^N is the set of nodes w.r.t. skeletal joints,  is a (weighted) adjacency matrix and  is a matrix of graph signals/attributes. According to the body bones between joints, we define those connected edges, and simply assign them to 1. In addition, other strategies (, Gaussian kernel) may be used to produce the adjacency matrix. The signal matrix =[_1, _2, ⋯, _d]∈^N× d is supported on the node set , whose i-th component (or channel) _i represents a signal of all nodes. That means, the i-th node v_i∈ is assigned with a signal vector of d dimensions. For skeletal data, we define the signal of each joint with its 3D spatial coordinates, , _i·:→ (x_i, y_i, z_i). Thus, for a dynamic graph sequence of length T, we may formulate it as a stream of graphs (_1, _2, ⋯, _T), where _k=(, _k, _k) denotes the skeleton at the k-th time slice.








  §.§.§ Multi-scale Graphical Convolutional Kernels


In the standard CNN running on images, the receptive field may be conveniently defined as a local square spatial region, due to grid-shaped structure. So convolutional filtering on regular structures is accessible. On the contrary, the construction of convolutional kernels on graphs is intractable because the same filtering responses are required for homogeneous graph structures. Inspired by the graph theory <cit.>, we resort to the adjacency matrix , which expresses the connections between nodes. As ^k exactly records the k-path reachable nodes, so we can construct a k-neighbor receptive field by defining a k-order polynomial of , denoted as ψ_k(). Taking the simplest case, let ψ_k()=^k, which actually describes the k-hop neighbor nodes. In practice, we may replace  with the Laplacian matrix Ł to avoid the scale effect of matrix norm during the recursive inference (see the following model). Thus, for receptive fields of K scales, we define multi-scale convolutional filtering as

    = *f = ∑_k=0^K-1ψ_k(Ł)_k,

where ψ_k(Ł) expresses the receptive field of the k-th scale and _k∈^d× d' is the corresponding signal transformation. The computation of ψ_k(Ł) weightedly summarizes the information of all nodes within the k-scale receptive field, which thus is homogenous-invariant for graphs.



  §.§.§ Spatio-Temporal Graph Convolution


Inspired by the design philosophy of autoregressive moving average (ARMA) <cit.>, we construct the spatio-temporal graphical convolutional model as follows,

    _t+1   =  ∑_k=0^K_1-1ψ_k(Ł)_t_k + _t_0, 
    Ø_t+1   = _t+1 + ∑_k=1^K_2-1ψ_k(Ł)_t_k,

where ψ_k(·) is a receptive field function on the k-th scale, {_k∈^d'× d', _k∈^d× d'} are the signal transformation matrices with regard to the k-scale,
K_1 and K_2 are the number of kernels respectively in the temporal and spatial domain. In the above model, =[_1,⋯,_d'], Ø=[ø_1,⋯,ø_d'] can be respectively viewed as the hidden state and the output state. Along time slices, signals are recursively regressed with local convolutional kernels in Eqn. (<ref>), thus motion variations can be sequentially encoded. The output signals in Eqn. (<ref>) combine spatially convolutional graph signals as well as dynamic temporal signals. Moreover, each output signal ø_i is dependent on all input signals {_1,⋯,_d}. Specifically, when signals are independent on each other, and the spatio-temporal convolutional filters are separately operated on a channel of signals, then the dynamic graph convolution model can be written as

    _t+1   =  ∑_k=0^K_1-1ψ_k(Ł)_t(_k) + _t(_̌0), 
    Ø_t+1   = _t+1 + ∑_k=1^K_2-1ψ_k(Ł)_t(_̌k),

where _k = [w_k1, ⋯, w_kd], _̌k = [v_k1, ⋯, v_kd] are mapping parameters, and w_ki, v_ki are associated to the k-th scale of the i-th signal. Note here the dimension of output is assumed to be the input dimension. It is easy to extend into d'≠ d in the case of signal independency.



























  §.§.§ Frequency Domain Analysis


As the receptive field function ψ_k(·) is a polynomial expression, we can derive ψ_k(Ł)=ψ_k(())=ψ_k(()). Thus the frequency response of convolution filtering in Eqn. (<ref>) can be written as

    = ∑_k=0^K-1(ψ_k())_k .

In particular, if signals are independent as expressed in Eqn. (<ref>) and (<ref>), then we can obtain the frequency responses of graphical signals as = ∑_k=0^K-1(ψ_k()_̌k)⊙, each frequency response of which is formally  _ij=∑_k=0^K-1(v_kjψ_k(λ_i))_ij. Therefore, when signals are independent, the multi-scale convolutional filtering on graphs may be regarded as a (K-1)-order polynomial approximation of graph Flourier transform, if let H(λ_i)=∑_k=0^K-1v_kiψ_k(λ_i).

In the signal dependency case, we can decompose _k into a diagonal matrix multiplied by a general matrix, , _k = (_k)_k. Then the frequency response of convolution filtering in Eqn. (<ref>) can be written as
= ∑_k=0^K-1((ψ_k()_k)⊙)_k.
Therefore, in the general case of signal dependency, the calculation of frequency responses may be understood as two steps: (i) perform a polynomial approximation of GFT on each input signal, and (ii) transform multi-channel signals by the new mappings {_k}.




  §.§.§ Stability Analysis


For any sequence of graph realizations {_1, _2, ⋯}, we can prove the stability of the recursive model (Eqn. (<ref>)∼(<ref>)), which is summarized in the following theory.


Suppose the Laplacian matrix Ł has the eigenvalue decomposition Ł=, and ψ_k(·) is a k-order polynomial function satifying ψ_k(Ł)_2 ≤ 1. For the signal mappings {_0,⋯, _K_1-1}, if their diagonal elements are non-negative, i.e., W_k,ii≥ 0, and ∑_k=0^K_1-1_k_∞<1, the frequency response vec() in Eqn. (<ref>) will converge to

    lim_t→∞vec(_t)=· vec(_t-1), 
       = (-Γ_0^K_1(,))^-1Γ_0^1(,) +Γ_1^K_2(, ),

where Γ_a^b(,)=∑_k=a^b-1 (_k⊗ψ_k()) (similar for Γ_0^K_2).
Moreover, the transformation function  has an upper-bound:

    _2 < _0_∞/1-∑_k=0^K_1-1_k_∞ + ∑_k=1^K_2_k_∞.




As ψ_k(·) is a polynomial function, we can transform the recursive model (Eqn. (<ref>)∼(<ref>)) from spatial domain into frequency domain by using = and =,

    _t+1   =  ∑_k=0^K_1-1ψ_k()_t_k + _t_0, 
    _t+1   = _t+1 + ∑_k=1^K_2-1ψ_k()_t_k .

By using the abbreviated notation Γ_a^b(·,·) and ψ_0(·)=, we can derive a vertorized style of Eqn. (<ref>) as

    vec(_t+1)  
                 
                     = Γ_0^K_1(,)vec(_t) + Γ_0^1(,)vec(_t)
                   = ∑_τ=0^t((Γ_0^K_1(,))^τ)Γ_0^1(,)vec(_t) 
         +(Γ_0^K_1(,))^t+1 vec(_0).

Next we need to prove Γ_0^K_1(,)_2<1. The matrix Γ_0^K_1(,) is a block matrix of d'× d' blocks, each of which is a diagonal matrix. Let ^(ij) = ∑_k=0^K_1-1W_k,ijψ_k() denote the (i,j)-th block diagonal matrix, where W_k,ij denotes the (i,j)-th element of the matrix _k. As ψ_k() is a diagonal matrix, we denote  ψ_k()=([_k1,_k2,⋯,_kn]) for simplification.
Since ψ_k(Ł)_2 ≤ 1, we have _ki∈[-1,1]. And, since ∑_k=0^K_1-1_k_∞<1, we have

    Γ_0^K_1(,)_2 ≤max_i,r∑_j|^(ij)_rr|
     =max_i,r∑_j|∑_k=0^K_1-1W_k,ij_kr| 
                   ≤max_i,r∑_j ∑_k=0^K_1-1|W_k,ij| < 1.

Therefore, when t→∞, we can derive vec(_t+1) as

    vec(_t+1)
           = (-Γ_0^K_1(,))^-1Γ_0^1(,)vec(_t).

After a simple algebra calculation on Eqn. (<ref>), we can reach the conclusion of Eqn. (<ref>)∼(<ref>).

Now we prove the bound of the transformation function . First, we can derive the following two inequations,

    (-_ii)^-1_∞^-1 = (-∑_k=0^K_1-1 W_k,iiψ_k())^-1_∞^-1
       =min_l |1-∑_k=0^K_1-1W_k,ii_kl| > =1-|∑_k=0^K_1-1W_k,ii|,
       ∑_j≠ i_ij_∞=∑_j≠ i∑_k=0^K_1-1 W_k,ijψ_k()_∞
       = ∑_j≠ imax_l|∑_k=0^K_1-1 W_k,ij_kl|
            <∑_j≠ i∑_k=0^K_1-1 |W_k,ij|.

Note the above derivation uses the conditions W_k,ii≥ 0 and _kl∈[-1,1]. When ∑_k=0^K-1_k_∞<1, we can further have (-_ii)^-1_∞^-1>∑_j≠ i_ij_∞ for ∀ i=1,⋯, d'. So -Γ_0^K_1(,) is strictly block diagonal dominant (SBDD). According to Ahlberg-Nilson-Varah bound of SSDD matrix <cit.>, after a series of algebra derivations, we can reach the bound:

    (-Γ(,))^-1_∞  < 1/1-∑_k=0^K_1_k_∞.

Similarly, Γ_0^1(,)_∞≤_0_∞, Γ_1^K_2(, )_∞≤∑_k=1^K_2_k_∞.
Finally we obtain an upper bound of .


When signals are independent, , the recursive model takes Eqn. (<ref>)∼(<ref>), we can have the following corollary based on the above theory.

Suppose the Laplacian matrix Ł has the eigenvalue decomposition Ł=()=([λ_1,⋯,λ_n]), and ψ_k(Ł) is a k-order polynomial. If signals are independent, i.e., taking the recursive model of Eqn. (<ref>)∼(<ref>), for ∀ i=1,⋯,d, j=1,⋯, n, and w_ki≥ 0 and |∑_k=0^K_1-1w_kiψ_k(λ_j)|<1, then the frequency response  in Eqn. (<ref>) will converge to

    lim_t→∞_t+1=⊙_t, 
       = _̌0/1-∑_k=0^K_1-1ψ_k()_k+ ∑_k=1^K_2-1ψ_k()_̌k,

where -,⊙ respectively denote the elementwise division and multiplication.
Further, if |ψ_k(λ_j)|<1, and considering  is the elementwise multiplication, then for ∀ (i,j) term of , we have an upper-bound:

    _ij_2 < _̌0_∞/1-∑_k=0^K_1-1_k_∞ + ∑_k=1^K_2_̌k_∞.



According to Eqn. (<ref>) in Theory <ref>, let Υ_a^b(,)=∑_k=a^b-1ψ_k()_k, we can have

    _t+1
                  = ∑_τ=0^t((Υ_0^K_1(,))^τ)⊙Υ_0^1(,̌)⊙_t
            +(Υ_0^K_1(,))^t+1⊙_0.

Since |∑_k=0^K_1-1w_kiψ_k(λ_j)|<1, when t→∞, we can have

    _t+1=Υ_0^1(,̌)/1-Υ_0^K_1(,)⊙_t.

Further, Eqn. (<ref>) can be obtained after simple derivations. The upper-bound of  can also be easily obtained..


In practice we can take some normalization strategies on {Ł, } to make them satisfy these preconditions, in order to guarantee model stability.



  §.§.§ Deep Stacking


The above recursive convolutional model can be easily extended into a deep architecture. Taking the recursive model as one basic layer, we may stack it into a multi-layer network architecture, in which the output signal Ø at the bottom layer is used as the input of the top layer. With the increase of layers, the receptive field size of convolutional kernels can become larger, thus the topper layer can abstract more global information. As observed from our experiments, this deep architecture (named deep STGC) can improve the performance of skeleton based action recognition. Besides, we may insert the recursive model into other networks as one basic unit to form a mixture network, according to the requirement of solved problems.



§ EXPERIMENTS




We conduct experiments on four public skeleton based action datasets: Florence 3D <cit.>, HDM05 <cit.>, Large Scale Combined dataset <cit.> and NTU RGB+D <cit.>. To investigate the effectiveness of our model, we conduct extensive experiments with different configurations listed as follows:

  
  * STGC (w/o Ł) is the standard baseline without considering neighbor nodes, , only performing channel mapping;
  
  * STGC w/ Ł uses convolutional kernels with the receptive filed of one hop neighborhood;
  
  * STGC_K (indep.) is the multi-scale filtering version under the assumption of signal independency.
  
  * STGC_K (dep.) is the general multi-scale filtering version, where signals are dependent on each other.
  
  * Deep STGC_K (indep.) is the deep scheme by stacking STGC_K (indep.).
  
  * Deep STGC_K (dep.) is the deep scheme by stacking STGC_K (dep.)





 §.§ Datasets and Settings




  §.§.§ Florence 3D (Florence)
 This dataset was collected from a stationary Kinect, and each body skeleton was recorded with only 15 joints. It contains 215 action sequences of 10 subjects with 9 actions: wave, drink from a bottle, answer phone, clap, tight lace, sit down, stand up, read watch, and bow. Due to a few skeletal joints, some types of actions are difficult to distinguish, such as drink from a bottle, answer phone and read watch.
We follow the standard experimental settings to perform leave-one-subject-out cross-validation <cit.>.



  §.§.§ HDM05


This dataset was captured by using an optical marker-based Vicon system, and contains 2337 action sequences of 130 motion classes, which are acted by 5 non-professional actors named “bd", “bk", “dg", “mm" and “tr". Each skeleton data is represented with 31 joints. Until now, this dataset should involve the most skeleton-based action categories to the best of our knowledge. Due to the intra-class variations and large number of motion classes, this dataset is challenging in action recognition.
To compare with those previous literatures, we conduct two types of experiments by following two widely-used protocols. Firstly, we use two subjects “bd" and “mm" for training and the remaining three for testing <cit.>. Secondly, to fairly compare the current deep learning methods, we conduct 10 random evaluations, each of which randomly selects half of the sequences for training and the rest for testing <cit.>.



  §.§.§ Large Scale Combined (LSC)


This dataset combines nine publicly available datasets
<cit.>, and form a complex action dataset with 88 actions. As each individual dataset has its own characteristics in action execution manners, backgrounds, acting positions, view angles, resolutions, and sensor types, the combination of a large number of action classes makes the dataset more challenging in suffering large intra-class variation compared to each individual dataset.
Following Zhang  <cit.>, we conduct experiments using two standard settings, , random cross subject evaluation and random cross sample evaluation. For each action, half of the subjects/samples are randomly selected for training while the rest for testing.



  §.§.§ NTU RGB+D (NTU)


This dataset is collected by Kinect v2 cameras from different views. It consists of 56880 sequences for 60 distinct actions, including various of daily actions and pair actions performed by 40 subjects. The skeleton data is represented by 25 joints. As far as we know, this dataset is currently the largest skeleton-based action recognition dataset. The large intra-class and view point variations make this dataset great challenging. Meanwhile, a large amount of samples will bring a new challenge to the current skeleton-based action recognition methods.
We follow the two types of standard evaluation protocols <cit.>, , cross-view evaluation, cross-subject evaluation, to perform experiments.





 §.§ Implementation Details




  §.§.§ Data Processing


As skeletal data is usually captured from multi-view points and human actions are independent on the user coordinate system, we modify the origin of the coordinate system as the orthocenter of joints for each frame of skeleton, ,
=1/N∑_i=1^N_i, where _i∈^3 is a 3D coordinate of the i-th joint, N is the number of joints. Specially, for NTU, we preprocess the joint coordinates in a way similar to Shahroudy  <cit.>.
To enhance the robustness of model training, we perform data augmentation as widely used in previous deep learning literature <cit.>. Concretely, for each action sequence, we split the sequence into several equal sized subsequences, here 12 segments, and then pick one frame from each segment randomly to generate a large amount of training sequences. In addition, we randomly scale the skeletons by multiplying a factor in [0.98, 1.02] for the sake of the adaptive capability of scaling.



  §.§.§ Model Configuration


For the undirected attribute graph, we simply design edge connections according to human bones. If two joints are bridged with a bone, the edge weight is assigned to 1, otherwise 0. The signals of each node are set to its 3D coordinate. For the receptive field function, we use the simplest polynomial term, , ψ_k(Ł)=Ł^k, which represents k-hop neighbors. According to Theory <ref>, we normalize/clip Ł=Ł/λ_max (λ_max=2), _k,ii≥ 0, ∑_k|_k|< after the gradient update at each iteration, to make sure the model stability. The outputs of recursive model are concatenated into a softmax layer for classification. In the default case, the dimension of output signals is d'=32. For deep STGC, we empirically observe that stacking two layers is good enough to these datasets, thus we only employ two-layer network for Deep STGC. The outputs of each layer in deep STGC are 32, 64 dimensions. The most important factor of our model is the scale of convolutional kernels, which is analyzed in the next Section <ref>.












 §.§ Selection of Convolutional Kernel Size


In the STGC_K model, the parameters {K_1,K_2} control the receptive field sizes in temporal domain and spatial domain. When increasing K_1, K_2, the local filtering region will cover the farther hoping neighbors. To check the effect of different scales, we conduct an experiment on HDM05 dataset by searching the temporal kernel scale K_1 in {1,2,3} and the spatial kernel scale K_2 in {1, 2,4,6,8}. Considering the continuous convolution filtering is performed along time slices like a stacking CNN, in practice we should employ a smaller K_1 than K_2. The cross-comparison results are reported in Figure <ref>. The best performance is obtained at K_1=2 and K_2=6, which are used as the default parameters. Note that, K=1 means STGC(w/o Ł), which convolves only on the node itself without any neighbors. If K_1=1 and K_2 > 1, only spatial filtering is taken.  Conversely, K_1>1 and K_2 = 1 for only temporal filtering. We can observe that the spatial and temporal filtering together contribute the gain for action recognition.



 §.§ Verification of STGC Structure


For our proposed model STGC itself, we conduct a series of experiments with different configurations on four benchmark datasets. The results are summarized in Table <ref>. They include six configurations: STGC(w/o Ł), STGC(w/ Ł), STGC_K(indep.), STGC_K(dep.), deep STGC_K(indep.) and deep STGC_K(dep.). The standard baseline should be STGC(w/o Ł), which doesn't use any adjacent relationship, , implementing a multi-channel mapping like 1× 1 convolution on images. When introducing the convolutional kernel with 1-neighborhood, , STGC(w/ Ł), the performance is improved. As discussed in Section <ref>, we set K_1=2 and K_2=6 as default for multi-scale STGC_K(indep.), STGC_K(dep.), where the former is the version of independent signals while the latter is the general one. STGC_K(dep.) is slightly superior to STGC_K(indep.) due to the consideration of signal interaction. When extending STGC into the deep architecture of two layers, we can achieve the best performance on all four benchmark datasets.



 §.§ Comparisons with State-of-the-Art


We compare state-of-the-art methods on Florence, HDM05, LSC, NTU, respectively, which are summarized in Table <ref>, <ref>, <ref> and <ref>. As observed from these results, we have the following observations.

The proposed spatio-temporal graph convolution method is superior to the recent graph-based method <cit.> and LSTM-based methods <cit.>. As shown in Table <ref>, our STGC has a large improvement (more than 7%) in contrast to the graph-based work <cit.>. In principle, our STGC is very different from this work <cit.>, although graph is used for both. Our method falls into a recursively convolutional architecture, while the work <cit.> follows the conventional graph kernel matching technique. Also, different from those LSTM-based methods, which only model dynamics of sequences by revising LSTM, our method absorbs the success of convolutional filtering into a recursive learning with a theoretical guarantee.

Our proposed STGC improves the current state-of-the-art on most datasets. On the Florence dataset, our method achieves a nearly perfect performance 99.07%. On the current largest dataset NTU, under the same recursive idea, the performance is pushed to the higher 86.28% and 74.85%, from 82.80% and 74.40% for GCA-LSTM. Recently, the CNN-based methods <cit.> converted skeletons into images and then employed the sophisticated CNN feature extraction techniques by using the pre-training on ImageNet <cit.>. Even so, without the use of extra training data, our STGC is still more competitive over them.

Deep learning based methods are more effective than those shallow learning methods. The advanced nonlinear dynamic networks, variations of LSTM <cit.> and CNN-based models <cit.>, largely improve the action recognition performance, due to their robust representation ability. For the conventional matrix-based descriptors (, covariance or its variants), although the deep manifold learning strategies <cit.> are developed recently, the matrix-based representations limit their representation capability because the only second-order statistic relationship of skeletal joints is preserved, whereas first-order statistics is also informative <cit.>.

Different datasets have different performances. Florence 3D is the simplest dataset with 215 sequences and 9 action classes, thus most methods obtain a higher accuracy. The most difficult dataset should be the largest dataset NTU, which consists of 56880 sequences and covers various of daily actions and pair actions. The cross subject accuracy is still less than 80% due to various entangled actions.
Cross subject is more difficult than cross view or cross sample. The phenomenon is observed from Table <ref>, Table <ref>, and Table <ref> (the left/right column w.r.t cross subject/cross sample). It is easy to understand, in the cross subject task, more unforeseeable information exists in the testing set, compared to the other tasks.















§ CONCLUSION


We proposed a spatio-temporal graph convolution approach for combining the successes of local convolution filtering and the recursive learning power of autoregressive moving average. To locally filter on spatio-temporal structures, we introduced multi-scale graphical convolutional kernels, which were composed of the receptive field matrices defined by polynomials of adjacency matrix, and signal mappings. The multi-scale convolutional kernels were simultaneously performed on hidden states of sequences for encoding the motion variations and input state for extracting spatial graphical feature. In theory, we proved the convergency of the proposed model and provided an upper-bound. Moreover, we extended the basic model into a multi-layer deep architecture. We verified the representation ability of the proposed STGC and its deep version. We also demonstrated the improvements with STGC on four public skeletal datasets, including the current largest NTU RGB+D dataset. As a generic model, the proposed model may be generalized into many problems modeled by dynamic graphs, which will be one of our  future work.



§ ACKNOWLEDGEMENT


This work was supported in part by the National Basic Research Program of China under Grant 2015CB351704 and 2014CB349303, in part by the National Natural Science Foundation of China under Grant Nos. 61772276, 61602244, 61472187, 61231002 and 91420201, and program for Changjiang Scholars.


  aaai
  


