Tensor graph convolutional neural network
Tong Zhang Wenming Zheng Zhen Cui Yang Li


































































Tensor graph convolutional neural network
    Tong Zhang,
        Wenming Zheng, Member, IEEE,
        Zhen Cui
        and Yang Li
Tong Zhang and Yang Li are with the Key Laboratory of Child Development and Learning Science of Ministry of Education, and the Department
of Information Science and  Engineering, Southeast University, China. (e-mail: tongzhang@seu.edu.cn;yang_li@seu.edu.cn).
Wenming Zheng is with the Key Laboratory of Child Development and Learning Science of Ministry of Education, Research Center for Learning Science, Southeast University, Nanjing, Jiangsu 210096, China (e-mail: wenming_zheng@seu.edu.cn; xhzongyuan@seu.edu.cn). 
Zhen Cui is with the School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, China. (e-mail: zhen.cui@njust.edu.cn).



    December 30, 2023
===================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================






In this paper, we propose a novel tensor graph convolutional neural network (TGCNN) to conduct convolution on factorizable graphs, for which here two types of problems are focused, one is sequential dynamic graphs and the other is cross-attribute graphs. Especially, we propose a graph preserving layer to memorize salient nodes of those factorized subgraphs, i.e. cross graph convolution and graph pooling. For cross graph convolution, a parameterized Kronecker sum operation is proposed to generate a conjunctive adjacency matrix characterizing the relationship between every pair of nodes across two subgraphs. Taking this operation, then general graph convolution may be efficiently performed followed by the composition of small matrices, which thus reduces high memory and computational burden. Encapsuling sequence graphs into a recursive learning, the dynamics of graphs can be efficiently encoded as well as the spatial layout of graphs. To validate the proposed TGCNN, experiments are conducted on skeleton action datasets as well as matrix completion dataset. The experiment results demonstrate that our method can achieve more competitive performance with the state-of-the-art methods.




tensor graph convolutional neural network, parameterized Kronecker sum operation, recursive learning.














§ INTRODUCTION
















Graph aphs such as recommending system (the connection of users and goods). In fact, all these graphs can be easily represented as a composition of several subgraphs. In this paper, we will focus on how to perform more efficient graph convolution on this type of problem. Especially, here we refer to two classic tasks: skeleton-based action recognition and recommending system.

As a representative of dynamic sequence graphs, skeleton-based action recognition has become a hot topic of computer vision, and it draws wide attention in recent years due to its wide applications, e.g. video surveillance, games
console and robot vision. In previous literatures, various algorithms have been proposed <cit.> to deal with skeleton data based action recognition. Some of them just focus on modeling temporal evolution while  fail to well characterize the spatial dependencies among joints. Different from these algorithms, some other literatures attempt to model spatial structure by employing structure learning algorithms such as Riemannian network <cit.>, Lie-Net <cit.>, various types of recurrent neural network (RNN) <cit.> and graph based representation <cit.>. Among them,  graph representation proposed in <cit.> provides an efficient way for describing the irregular graph-structured skeleton data by modeling joints of skeleton as nodes of a graph. Further more, recent year, based on spectral graph theory, graph convolution neural network (GCNN)  is proposed in <cit.> for irregular data as an alternative algorithm to CNN and shows promising performance. Although graph convolution has been used for sequence data, it more constructs a graph for each frame, whilst ignore temporal correlation. In contrast, we conduct a large spatio-temporal graph convolution by simultaneously model spatial and temporal relationship of different nodes.

Another representative of artificial intelligence application based on cross attribute factorized graphs is recommending system,  which can be formed as matrix completion problem and has been investigated in previous literatures  <cit.>. In this task, two separable graphs of different attributes, i.e. column graphs and row graphs,  are provided representing
similarity of users and items and  were shown beneficial for the performance of recommender systems.  <cit.> applies graph convolution in this task considering both users and item graphs from the Fourier transform respective and achieves the state-of-the-art performance. However, quite different from <cit.>, we propose to conduct cross graph convolution for this task aiming to optimally model the relationship between every pair of nodes from two graphs.

In this paper, taking the two representative tasks, we explore convolution on factorizable graphs. More generally, we proposed a tensor graph convolutional neural network (TGCNN) to deal with the problem. For sequential dynamic graphs, especially, we leverage recursive mechanism on consecutive tensor convolution to overcome the problem of computational explosion. Therein, a graph preserving layer is proposed to recursively optimize both previous encoded spatio-temporal graph and the successively input subgraph. In each recursive step, the salient nodes in previous subgraphs are preserved and further connected to those in current input graph, so that the graph preserving layer is able to globally model all those nodes in factorized graphs. To this end, two operations are employed in the graph preserving layer, cross graph convolution (for building cross graph relationship) and graph pooling (for nodes selecting). Specifically, for cross graph convolution, we design a novel parameterized Kronecker sum operation to learn an optimal conjunctive graph. During derivation, the property of Kronecker product is utilized so that  graph filtering can be conduced on matrices in smaller size, which thus may avoid high memory and computational costs. Following the conjunctive cross graph relationship, graph pooling is used to choose an optimal subset of salient nodes for next recursive process.


To evaluate the proposed TGCNN, we conduct experiments on two large scale action datasets named NTU RGB+D (NTU) dataset <cit.> and the Large Scale Combined (LSC) dataset <cit.>. Moreover, to test the generalization ability of the proposed cross graph convolution on graph structural data, we also conduct an extensive experiment on a matrix completion dataset <cit.> named Synthetic ‘Netflix’ dataset. The experimental results show that our method outperforms those state-of-the-arts.

In summary, our main contributions are three folds:


  * we propose a novel tensor graph convolutional neural network which is able to globally learn an optimal graph from multiple factorized subgraphs through a recursive learning process.


  * we design cross graph convolution to efficiently encode relationship of each pair of nodes across two subgraphs, and efficiently derive in tensor space by utilizing the property of Kronecker product.

  * we experimentally validate the effectiveness of our method on both action recognition and matrix completion datasets, and report the state-of-the-art results.






§ PRELIMINARY OF SPECTRAL FILTERING

Let  𝐬 = [s_1, s_2, ..., s_n]^T be a signal where each element represents a vertex in a graph, and 𝐀 be the corresponding adjacency matrix, then the spectral filtering on 𝐬 can be formulated as:
  
    𝐬̃ = g_θ(𝐋)𝐬 = g_θ(𝐔Λ𝐔^T)𝐬=𝐔g_θ(Λ )𝐔^T 𝐬

where
  
    𝐋 = 𝐈 - 𝐃^1/2𝐀𝐃^1/2 =  𝐔^TΛ𝐔,  g_θ(Λ)=∑_k=0^K-1θ_kΛ_k

In Eqn. <ref>, 𝐃∈ℝ^n × n is a diagonal degree matrix  with the diagonal elements calculated as D_ii = ∑_jA_ij.

Furthermore, according to  <cit.>, 𝐋 can be replaced by 𝐀 as they have the same Fourier basis.
Then after substituting Eqn. <ref> to Eqn. <ref> and replacing  𝐋 with 𝐀, spectral filtering can be rewritten as follows:
  
    𝐬̃= ∑_k=0^K-1θ_k 𝐀^k 𝐬




§ TGCNN ARCHITECTURE

 The whole architecture of the proposed TGCNN is shown in Fig. <ref>, in which the key component  is the graph preserving layer.  In following subsections, we first describe the two  key operations of graph preserving layer, i.e. cross graph convolution and  graph pooling.  Then we show the whole recursive learning process of graph preserving layer.



 §.§ Cross graph convolution

 Cross graph convolution consists of two main steps, i.e. cross graph construction and spectral filtering. Cross graph construction aims to build conjunctive graph between  two subgraphs, where each pair of nodes from  these two subgraphs are well modeled.  In cross graph construction process,  let 𝐱∈ℝ^n_1 and 𝐲∈ℝ^n_2 be two signals and 𝐀_x ∈ℝ^n_1 × n_1, 𝐀_y ∈ℝ^n_2 × n_2 be the corresponding adjacency matrices, then the conjunctive signal 𝐒=[𝐬_1, 𝐬_2] is defined as follows:

    𝐒 = [𝐬_1, 𝐬_2] =  f(𝐱, 𝐲)                            
              = [ 𝐱⊗ (  1^n_2),  ( 1^n_1) ⊗𝐲;                            ],

where

    𝐬_1, 𝐬_2 ∈ℝ^n_1* n_2, 𝐒∈ℝ^(n_1* n_2)× 2.

And according to spectral graph theory, the adjacency matrix of 𝐒, denoted as 𝐀∈ℝ^(n_1*n_2) × (n_1*n_2), should correspondingly describe the similarities between each pair of nodes in 𝐒. In previous research, Kronecker sum operation is proposed which can be used for describing conjunctive  similarities of two given subgraphs:

    𝐀= 𝐀_x ⊕𝐀_y= 𝐀_x ⊗𝐈^n_2+  𝐈^n_1⊗𝐀_y.

 However, due to the complexity of signals, the  conjunctive adjacency matrix generated by  classic Kronecker sum operation may not well fit 𝐒. For this reason, we parameterize the  Kronecker sum operation expecting to learn a optimal conjunctive adjacency matrix from 𝐀_x, 𝐀_y. The new operation,  denoted as ⊕_p, is named as parameterized  Kronecker sum operation which is defined as follows:
 
    𝐀= 𝐀_x ⊕_p𝐀_y= 𝐀_x ⊗𝐈_λ_1+  𝐈_λ_2⊗𝐀_y, 
    where 𝐈_λ = diag(λ),  λ_1 ∈ℝ^n_2, λ_2 ∈ℝ^n_1.

In Eqn. <ref>,  λ_1, λ_2 are both trainable vectors.

Then, we conduct cross graph convolution by applying spectral filtering on 𝐒 and 𝐀:

    𝐒̃ = 𝐇_θ(𝐀, 𝐒) = ∑_k=0^Kθ_k 𝐀^k 𝐒         
                = [∑_k=0^Kθ_k𝐀^k 𝐬_1, ∑_k=0^Kθ_k𝐀^k𝐬_2 ].

 In this process, the key step is to calculate the k-th order polynomials of the adjacency matrix. Formally, the 1-st order polynomial on 𝐬_1 can be calculated as follows:

    𝐀𝐬_1 =   (𝐀_x ⊗𝐈_λ_1+ 𝐈_λ_2⊗𝐀_y)𝐬_1                           
               =  (𝐀_x ⊗𝐈_λ_1)𝐬_1 + (𝐈_λ_2⊗𝐀_y)𝐬_1                     
              = vec(𝐈_λ_1mat(𝐬_1)𝐀_x^T)+vec(𝐀_y mat(𝐬_1) 𝐈_λ_2)

where vec(·) denotes the vectorization of a matrix by stacking its columns into a single column vector, and mat(·) is the reverse process of vec(·) which transforms a single column vector into a matrix. So, we also have the following equation:

    mat(𝐀𝐬_1)=𝐈_λ_1mat(𝐬_1)𝐀_x^T+𝐀_y mat(𝐬_1)𝐈_λ_2.

In above equations, Eqn. <ref> is transformed to Eqn. <ref> by utilizing the property of Kronecker product. Then, the 2-nd order polynomial on 𝐬_1 can be further calculated as:

    𝐀^2𝐬_1 = 𝐀(𝐀𝐬_1)                                               
    
                                                      =(𝐀_x ⊗𝐈_λ_1+𝐈_λ_2⊗𝐀_y)(𝐀𝐬_1)                    
    
                                                      = vec(𝐈_λ_1mat(𝐀𝐬_1)𝐀_x^T+𝐀_y mat(𝐀𝐬_1)𝐈_λ_2)

Based on the results of 1-st and 2-nd order polynomials,  the K-th order polynomial on 𝐬_1 can be deduced, which can be written as:

    𝐀^K𝐬_1 = 𝐀(𝐀^K-1𝐬_1)                                         
    
                                                      = vec(𝐈_λ_1mat(𝐀^K-1𝐬_1)𝐀_x^T+𝐀_y mat(𝐀^K-1𝐬_1)𝐈_λ_2).

By utilizing the property of Kronecker product,  the graph convolution on 𝐀, which needs to calculate polynomials on matrices in size (n_1*n_2) × (n_1*n_2),  is transformed to calculate the matrix product on two matrices in much smaller sizes,  i.e. (n_1 × n_1) and (n_2 × n_2) respectively. Correspondingly, the computational cost also decreases from (n_1*n_2) × (n_1*n_2) to (n_1*n_1) + (n_2*n_2), which effectively relieves the computational burden.

Moreover, to further improve the computation efficiency,  the vectorization operation denoted as  vec(·) and its reverse operation  mat(·) need not be conducted frequently  by transforming Eqn. <ref> to the following format:
 
    𝐒̃= [vec(∑_k=0^Kθ_kmat(𝐀^k 𝐬_1)), vec(∑_k=0^Kθ_kmat(𝐀^k𝐬_2) )]




 §.§ Graph pooling

As cross graph convolution results in a large graph of n1*n2 nodes from two factorized graphs where the  total number of nodes is n1+n2,  there may be some irrelevant nodes contributing little to action recognition. This part of nodes increase computation cost and also may degrade the performance. To reduce the disturbances of these irrelevant nodes as well as reduce the graph size, a projecting matrix is utilized to weight nodes so that an optimal subset of salient nodes can be effectively selected.

Formally, given the nodes denoted as  𝐒 and it's corresponding adjacency matrix 𝐀, the graph pooling process can be described as follows:
 
    p(𝐒)= 𝐖𝐒,     
    
    p(𝐀)=𝐖𝐀𝐖^T.

where 𝐖 is the parameter to be solved. The larger the matrix element is, the more important the corresponding weighted node is for action recognition. This graph pooling operation not only further promotes the performance of TGCNN, but also avoids serious graph expansion during the recursive learning process  introduced in the following subsection in detail.



 §.§ Recursive learning process

 The recursive learning process aims to transform the factorized graph modeling from processing all nodes at  one time to recurrent convolution on two subgraphs step by step, which effectively relieves the computational burden. Formally, let  𝐀_1, 𝐀_2, ...,  𝐀_T denote T factorized subgraphs and 𝐒_1, 𝐒_2, ...,  𝐒_T be their corresponding signals,  based on cross graph convolution and graph pooling, the recursive learning process can be formulated as follows:

    𝐒^c_n = f(𝐒_n, 𝐒̃_n-1),            
    𝐀^c_n = 𝐀_n ⊕_p Ã_n-1,       
    𝐒̃_n =  p(𝐇_θ(𝐀^c_n, 𝐒^c_n)),     
    Ã_n = p(𝐀^c_n),               
    𝐎_n =  σ_1(𝐖_co𝐒̃_n+𝐛_co).

In above equations,  𝐀^c_n and  𝐒^c_n represent the conjunctively constructed adjacency matrix and signal respectively, which are  generated from the current input and previous preserved graphs.  Ã_n and  𝐒̃_n  denote the preserved adjacency matrix and the filtering signal at the n-th recursive step, and 𝐎_n is the n-th output feature. 𝐖_co and 𝐛_co are learnable variables for learning an output feature from the filtering signal, and σ_1(·) is a non-linear activation function which is used for endowing flexibility to the graph preserving layer.

Eqn. <ref>-<ref> cooperate to achieve global learning on all input factorized subgraphs. Given successive factorized graphs, the nodes between the preserved and input graphs are jointly modeled  through cross graph convolution (Eqn. <ref>-<ref>). Then, based on this,  graph pooling is applied which acts as a memory unit to remember those salient nodes. These two operations are recursively conducted so that the learning process makes TGCNN be inherently deep as the previous input graphs are connected with current input one. And thus, the output features are able to describe the global graph structure.



 §.§ The loss functions

For action recognition, the output features of graph preserving layer  are further passed through a full connection layer and a softmax layer. Then cross entropy loss is employed for TGCNN training, which can be defined as follows:

    E =-∑_i=1^N∑_c=1^Cτ(y_i,c) ×log P(c|𝐒_i1, ⋯, 𝐒_iT)

in which

    τ(y_i,c)=
          
             1,      y_i=c; 
    
             0,

where E denotes the cross entropy loss calculating the mean negative logarithm value of the prediction probability of the training samples, N denotes the number of the training samples, 𝐒_i1, ⋯, 𝐒_iT represent input factorized graphs of i-th training sample and y_i is the corresponding label.

Besides action recognition, to test the generalization ability of the proposed cross graph convolution on graph structural data, we also conduct an extensive experiment on a matrix completion dataset <cit.>. In matrix completion, only two factorized subgraphs are provided instead of multiple graphs. So, this task can be treated as a simplified application case of our TGCNN. As the algorithm in <cit.> achieves the current best performance, for a fair comparison, we embed our cross graph convolution into the framework of <cit.> and employ the same loss function as <cit.>, which is commonly employed in matrix completion algorithms.



§ EXPERIMENTS

 We evaluate  our TGCNN on two action recognition datasets named NTU RGB+D (NTU) dataset <cit.> and  Large Scale Combined (LSC) dataset <cit.>. Besides, to test the generalization ability to graph structural data, we also conduct an extensive experiment on a matrix completion dataset named Synthetic ‘Netflix’ dataset <cit.>, in which two factorized graphs are provided describing the relationship among users and items respectively. In the following subsections, we firstly introduce these three datasets, then we show the implemental details including the preprocessing of action recognition datasets. Finally, we compare the experimental results with the state-of-the-art methods.



 §.§ Datasets

In NTU dataset, there are 56880 RGB+D video samples executed by 40 different human subjects whose ages are in the range from 10 to 35.  Three synchronous  Microsoft Kinect v2 sensors are used for collecting various modalities of signals from three different horizontal angles, where the modalities include RGB videos, depth sequences, skeleton data and infrared frames. Specifically, for skeleton data, the human skeleton is represented by 3D locations of 25 major body joints. This dataset is great challenging due to its large amount of samples, multiple view points and  intra-class variations.

The LSC dataset is an integrated dataset created by combing nine existing public datasets. In this dataset, there are 4953 video sequences containing red, green and blue (RGB) video and depth information. These sequences contain 94 action classes which are performed by 107 subjects in total.  As these video sequences come from different individual datasets, the variations with respect to subjects, performing manners and backgrounds are very large. Moreover,  there is large difference among the number of samples of each action. All these factors, i.e. the large size, the large variations and the data imbalance for each class, make this dataset challenging for recognition.

Synthetic ‘Netflix’ dataset is  frequently used in matrix completion task to evaluate different algorithms. In this dataset, the row axis represents different item (e.g. movie) while  the column represents different users.  Thus the value of each element  shows whether a user would like an item or not.  When creating this dataset,  the matrix  is generated by satisfying certain assumptions, e.g. low rank property and smoothness along rows and columns. Thus,  there is strong communities structure in the generated user and item graphs. The advantage of  this dataset is that it enables the behaviours of different algorithms  be well studied  in controlled settings.





 §.§ Preprocessing on skeleton data

 The preprocessing of skeleton data aims to eliminate noise and also make the model be robust to different kinds of variations, e.g. body orientation variation and body scale variation. This process is done by the following three steps:

(i) The action sequences are first split to a fixed number of subsequences, and then one frame is chosen from each subsequence so that the generated sequences contain the same number of frames.

(ii) The skeletons are randomly scaled with different factors ranging in [0.95, 1.05] so that the adaptive scaling capacity of the model can be improved.

(iii)  During training stage, the skeletons randomly are rotated  along x,y and z axis with angles ranging in [-45, 45], which makes the model be robust to orientation variation.



 §.§ Experiment on NTU dataset

The experiments on NTU dataset are conducted following two different protocols,  named cross subject  and cross view protocols respectively, in <cit.>. For cross subject protocol,  samples are  split to training and testing sets according to subjects' ID numbers. Under this protocol,  the split training and testing sets contain 40320 and 16560 samples respectively where the samples in each set are conducted by 20 subjects. For cross view protocol, there are 37920 and 18960 samples in training and testing sets respectively. Among them, samples in training set are captured by  cameras #2 and #3 while the samples in testing set are captured by camera #1.

The main parameters in TGCNN are the polynomial order denoted as K, the numbers of preserved nodes and the dimension of  output features. For both protocols, K is set to be 2, the numbers of preserved nodes are both 50 and the dimension of output feature is 128.  Table <ref> shows the comparison results on NTU dataset. The proposed framework is compared with various the-state-of-the-art methods, including different kinds of recurrent neural networks (RNNs),  hierarchical bidirectional recurrent neural networks (HBRNN) <cit.>,  part-aware LSTM (P-LSTM) <cit.>, spatio-temporal LSTM (ST-LSTM) <cit.>, , and geometric features LSTM (GF-LSTM) <cit.>. For both protocols, our algorithm achieves the best performance.






 §.§ Experiment on Large Scale Combined dataset

We conduct experiments on LSC dataset by following two different protocols employed in <cit.>. For the first protocol named  Random Cross Sample (RCSam) using data of 88 action classes,  half of the samples of each class are randomly selected as training data while the rests are used as testing data.  For the second protocol named  Random Cross subject (RCSub) using data of  88 action classes,  half of the subjects are randomly selected as training data and the rest subjects are used as testing data. In both protocols, only skeleton data are used for recognition. Due to the imbalance of samples in each class,  the values of  precision and  recall are employed for evaluating the performance instead of accuracy.



The parameter settings for both RCSam and RCsub protocols are the same: the numbers of nodes in preserving layer are set to be 30 while the dimension of output feature is set to be 80, and K  is set to be 2.  The comparisons on LSC dataset are shown in Table <ref>. Except the recall value in RCSam protocol, our algorithm outperforms the previous state-of-the-art methods.



 §.§ Experiment on Synthetic ‘Netflix’ dataset






We follow the protocol employed in <cit.> to evaluate the performance of our TGCNN on Synthetic ‘Netflix’ dataset.  Under this protocol, a part of  chosen values of users are first eliminated, and  the task of algorithm  is to recovering the missing values of this matrix based on the given  fraction of  entries. At last, root mean squared (RMS) error is employed to evaluate the difference between the recovered matrix and the ground truth. The smaller value of RMS error means the  better performance.

The results of different matrix completion methods are reported in Table <ref>, along with their theoretical complexities. Algorithms including geometric matrix completion
(GMC) <cit.>, recurrent graph CNN (RGCNN) <cit.>, separable recurrent graph CNN (sRGCNN) <cit.> and graph regularized alternating least squares (GRALS) <cit.>, are compared with our TGCNN. As it is shown,  our TGCNN model achieves the best accuracy which demonstrates the generalization ability of TGCNN for graph structural data.





 §.§ Analysis of TGCNN




As  TGCNN achieves promising performance, it is meaningful to verify how much the novel proposed operations, e.g. cross graph convolution and parameterized Kronecker sum, improve the performance of the network, and also how the parameter setting influences the  result. For these purposes, several additional experiments are respectively conducted which are listed as follows:


  (1) TGCNN vs isolate graph CNN (IGCNN). To evaluate the effectiveness of the proposed recursive process, we conducted additional experiments on LSC datsets  comparing the results of TGCNN with IGCNN, where in IGCNN the graph filtering is only conducted  on each isolate graph (Table <ref>).

  (2) Parameterized Kronecker sum  vs classic Kronecker sum. To see how much improvement the parameterized Kronecker sum operation brings, we conducted experiments on both LSC and  Synthetic ‘Netflix’ dataset to compare the performance (Table <ref>).

  (3) Setting different orders of polynomials. We conduct additional experiments on LSC dataset following RCSub protocol to see how different  polynomial orders influent the performance  (Table <ref>).

From the results we can have the following observations:


  (i) TGCNN outperforms IGCNN which verifies the effectiveness of the recursive learning process, which globally learns the additional cross graph relationship comparing to IGCNN.

  (ii) The parameterized Kronecker sum operation further promotes the performance comparing to the classic one. This indicates that through parameterized Kronecker sum operation, the constructed conjunctive cross graph better fits the corresponding conjunctive signal.

  (iii) The value of polynomial order influences the performance of graph filtering.  On LSC dataset, the best precision is achieved by setting K to be 2 while the best recall is achieved when K is set to be 5.




§ CONCLUSION

In this paper, we proposed a novel framework named  TGCNN  to globally model those subgraphs factorized from a large graph. For this purpose, we  propose a  recursive learning process  on graph by specifically designing a novel graph preserving layer. Serving as a memory unit, this graph preserving layer memorizes those salient nodes of successively input graphs, where the  memory function is achieved through applying  novelly designed cross graph convolution and graph pooling. Specifically, cross graph convolution well models the relationship of each pair of nodes across graph and can be efficiently conducted. Besides, the proposed parameterized Kronecker product learns an optimal conjunctive adjacency matrix which further promotes the performance.  Comprehensive experiments conducted on  action recognition and matrix completion datasets verify the competitive performance of our TGCNN.





























































1

amor2016action
B. B. Amor, J. Su and A. Srivastava, Action recognition using rate-invariant analysis of skeletal shape
  trajectories. 0.4emIEEE transactions on pattern analysis and machine intelligence, vol. 38, no.1, pp. 1-13, 2016.

bruna2013spectral
J. Bruna, W. Zaremba, A. Szlam, and Y. LeCun, Spectral networks and locally connected networks on graphs. 0.4emarXiv preprint arXiv:1312.6203, 2013.

chaudhry2013bio
Rizwan Chaudhry, Ferda Ofli, Gregorij Kurillo, Ruzena Bajcsy, and René
  Vidal, Bio-inspired dynamic 3d discriminative skeletal features for human
  action recognition. 0.4emProceedings of the IEEE Conference on Computer Vision and
  Pattern Recognition Workshops, pp. 471-478, 2013.

defferrard2016convolutional
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst, Convolutional neural networks on graphs with fast localized spectral
  filtering. 0.4emAdvances in Neural Information Processing Systems, pp.
  3844-3852, 2016.

du2015hierarchical
Yong Du, Wei Wang, and Liang Wang, Hierarchical recurrent neural network for skeleton based action
  recognition. 0.4emProceedings of the IEEE conference on computer vision and
  pattern recognition, pp. 1110-1118, 2015.

huang2017riemannian
Zhiwu Huang and Luc J Van Gool, A riemannian network for spd matrix learning. 0.4emAAAI, vol.2, pp. 1-6, 2017.

huang2017deep
Zhiwu Huang, Chengde Wan, Thomas Probst, and Luc Van Gool, Deep learning on lie groups for skeleton-based action recognition. 1em plus
  0.5em minus 0.4emIn Proc. IEEE Conference on Computer Vision
  and Pattern Recognition, pp. 6099-6108, 2017.

kalofolias2014matrix
Vassilis Kalofolias, Xavier Bresson, Michael Bronstein, and Pierre
  Vandergheynst, Matrix completion on graphs. 0.4emarXiv preprint arXiv:1408.1717, 2014.

kipf2016semi
Thomas N Kipf and Max Welling, Semi-supervised classification with graph convolutional networks. 0.4emarXiv preprint arXiv:1609.02907, 2016.

li2017action
Chaolong Li, Zhen Cui, Wenming Zheng, Chunyan Xu, Rongrong Ji, and Jian Yang, Action-attending graphic neural network. 0.4emarXiv preprint arXiv:1711.06427, 2017.

liu2016spatio
Jun Liu, Amir Shahroudy, Dong Xu, and Gang Wang, Spatio-temporal lstm with trust gates for 3d human action
  recognition. 0.4emEuropean Conference on Computer Vision, pp. 816-833,
  Springer, 2016.

monti2017geometric
Federico Monti, Michael M Bronstein, and Xavier Bresson, Geometric matrix completion with recurrent multi-graph neural
  networks. 0.4emarXiv preprint arXiv:1704.06803, 2017.

ofli2014sequence
Ferda Ofli, Rizwan Chaudhry, Gregorij Kurillo, René Vidal, and Ruzena
  Bajcsy, Sequence of the most informative joints (SMIJ): A new representation
  for human skeletal action recognition. 0.4emJournal of Visual Communication and Image Representation, vol. 25, no.1, pp. 24-38, 2014.

oreifej2013hon4d
Omar Oreifej and Zicheng Liu, Hon4d: Histogram of oriented 4d normals for activity recognition from
  depth sequences. 0.4emComputer Vision and Pattern Recognition, pp. 716-723,  2013.

rao2015collaborative
Nikhil Rao, Hsiang-Fu Yu, Pradeep K Ravikumar, and Inderjit S Dhillon, Collaborative filtering with graph information: Consistency and
  scalable methods. 0.4emAdvances in neural information processing systems, pp.
  2107-2115, 2015.

sandryhaila2013discrete
Aliaksei Sandryhaila and José MF Moura, Discrete signal processing on graphs. 0.4emIEEE transactions on signal processing, vol. 61, no.7, pp. 1644-1656, 2013.

shahroudy2016ntu
Amir Shahroudy, Jun Liu, Tian-Tsong Ng, and Gang Wang, Ntu rgb+ d: A large scale dataset for 3d human activity analysis. 0.4emProceedings of the IEEE Conference on Computer Vision and
  Pattern Recognition, pp. 1010-1019, 2016.

vemulapalli2014human
Raviteja Vemulapalli, Felipe Arrate, and Rama Chellappa, Human action recognition by representing 3d skeletons as points in a
  lie group. 0.4emProceedings of the IEEE conference on computer vision and
  pattern recognition, pp. 588-595, 2014.

wang2016graph
Pei Wang, Chunfeng Yuan, Weiming Hu, Bing Li, and Yanning Zhang, Graph based skeleton motion representation and similarity measurement
  for action recognition. 0.4emIn Proc. European Conference on Computer Vision, pp. 370-385.
  Springer, 2016.

xia2012view
Lu Xia, Chia-Chih Chen, and JK Aggarwal, View invariant human action recognition using histograms of 3d
  joints. 0.4emIEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 20-27, 2012.

zhanglarge
Jing Zhang, Wanqing Li, Pichao Wang, Philip Ogunbona, Song Liu, and Chang Tang, A large scale rgb-d dataset for action recognition.

zhang2017geometric
Songyang Zhang, Xiaoming Liu, and Jun Xiao, On geometric features for skeleton-based action recognition using
  multilayer lstm networks. 0.4emIn Proc. IEEE Winter Conference on Applications of Computer Vision, pp. 148-157, 2017.














































