An Improved Graph Pooling Network for Skeleton-Based Action Recognition
Cong Wu Xiao-Jun Wu Tianyang Xu Josef Kittler







  Jiangnan University
  Wuxi
  China
congwu@stu.jiangnan.edu.cn




  Jiangnan University
  Wuxi
  China
wu_xiaojun@jiangnan.edu.cn



  Jiangnan University
  Wuxi
  China
tianyang_xu@163.com



  University of Surrey
  Guildford
  United Kingdom
j.kittler@surrey.ac.uk



Pooling is a crucial operation in computer vision, yet the unique structure of skeletons hinders the application of existing pooling strategies to skeleton graph modelling. In this paper, we propose an Improved Graph Pooling Network, referred to as IGPN. The main innovations include: Our method incorporates a region-awareness pooling strategy based on structural partitioning. The correlation matrix of the original feature is used to adaptively adjust the weight of information in different regions of the newly generated features, resulting in more flexible and effective processing. To prevent the irreversible loss of discriminative information, we propose a cross fusion module and an information supplement module to provide block-level and input-level information respectively. As a plug-and-play structure, the proposed operation can be seamlessly combined with existing GCN-based models. We conducted extensive evaluations on several challenging benchmarks, and the experimental results indicate the effectiveness of our proposed solutions. For example, in the cross-subject evaluation of the NTU-RGB+D 60 dataset, IGPN achieves a significant improvement in accuracy compared to the baseline while reducing Flops by nearly 70%; a heavier version has also been introduced to further boost accuracy.



<ccs2012>
   <concept>
       <concept_id>10010147.10010178.10010224.10010225.10010228</concept_id>
       <concept_desc>Computing methodologies Activity recognition and understanding</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
   <concept>
       <concept_id>10003033.10003034.10003035.10003036</concept_id>
       <concept_desc>Networks Layering</concept_desc>
       <concept_significance>300</concept_significance>
       </concept>
 </ccs2012>


[500]Computing methodologies Activity recognition and understanding
[300]Networks Layering





An Improved Graph Pooling Network for Skeleton-Based 
Action Recognition
    Josef Kittler
    April 29, 2024
========================================================================








§ INTRODUCTION

Recent studies have verified the inherent advantages of Graph Convolutional Networks (GCNs) in the representation of non-Euclidean structured data <cit.>, with GCN-based methods achieving leading performance in various fields <cit.>.
Yan et al. <cit.> first proposed to construct spatial-temporal graph to model skeleton sequences according to their natural arrangement. Refer to this paradigm, subsequent methods <cit.> have made significant improvements in performance.
However, only global pooling is used before the final classifier in most methods, leading to the redundancy of homogeneous information in graph modelling <cit.>.
Simultaneously, pooling is a conventional feature aggregation technique in Convolutional Neural Networks (CNNs), which typically integrates local information obtained after convolutional layers to produce more abstract feature representations. 
This operation reduces redundant information, lowers the computational cost, prevents over-fitting, and enhances the model's generalisation ability, while also increasing the integration of features and obtaining higher-level abstract representations through aggregation.
The pooling operation is not only justified theoretically <cit.>, 
but also has been shown to significantly improve performance in many computer vision tasks <cit.>.

Skeleton graph pooling strategy has recently gained significant attention in research.
<cit.> introduced novel pooling and unpooling operators within different homeomorphic skeletons. 
<cit.> divided the skeleton into regions based on physical arrangement and performed continuous pooling operations in each local region.
Those approaches bring higher computational efficiency while achieving higher recognition accuracy.
However, both methods rely on fixed manual pooling rules, which may remove useful information and limit their flexibility.
<cit.> introduced triplet pooling loss to facilitate adaptive clustering of skeleton graph nodes and automatically generate new embedded matrix representations. This approach introduces great flexibility, making the pooling process a trainable method. However, it does not explicitly consider the structural-semantic information.
In addition, existing pooling methods tend to force the network to discard essential but not the most critical nodes gradually.
However, the accumulation of multiple pooling operations can significantly affect the final performance. Hence, it is necessary to preserve and strengthen the discriminative and heterogeneous features while discarding unnecessary homogeneous features.

To address the limitations of existing pooling networks that rely on either fixed manual rules or excessive flexibility while ignoring the inherent structure, we propose a trainable structure pooling operation that strikes a balance between flexibility and structural preservability.
Specifically, we define the pooling area according to the physical structure of the skeleton, then calculate the contribution between each feature point in the pooling area and the entire graph representation according to the embedding similarity. After that, we adjust the proportion of the current feature point in the pooled features by constructing a trainable assign matrix. 

With the accumulation of multi-layer pooling operations, the loss of information will only intensify without remediation, the network capacity also will be surplus.
Here we propose block-based and network-based feature enhancement strategies, namely cross fusion block and information supplement module.
In cross fusion block, based on the original pooling network, we design another parallel operation that directly performs graph operations and then fuses the features obtained from the two parallel structures after alignment. 
In this way, feature fusion of different granularities is performed to supplement and strengthen the information of features.
In information supplement module, we decompose the skeleton sequence into position and vector-based features, then project them into common space through graph embedding for sufficient alignment, and combine them to obtain an efficient representation through feature fusion.


The conducted ablation experiments demonstrate our motivation and the effectiveness of the proposed method, and the results combined with various mainstream methods consistently show that our method can significantly reduce computational overhead while improving model performance. 
As shown in Table <ref>, 
the proposed improved graph pooling operation 
can can greatly reduce training overhead while ensuring improvement in accuracy performance.

Overall, our innovations can be summarised as follows:


  * Aiming at the structural properties of the skeleton sequences, we propose a structure pooling strategy with region awareness, which increases the pooling flexibility while maintaining the skeleton structural characteristics.

  * We use the cross fusion module and information supplement module to provide block-level and input-level information respectively, thus avoiding the loss of a large amount of information and the under-saturation of the network.

  * We conduct comprehensive experiments on NTU-RGB+D (60&120) <cit.>, UWA3D Multiview Activity II <cit.> datasets respectively, the  experimental findings provide full validation of our strategies.




§ RELATED WORK



 §.§ Skeleton-based Action Recognition

Non-GCN Methods. 
Due to the particularity of the structure of skeleton features, existing strategies often focus on their essential physical characteristics to carry out targeted modelling. 
Traditional methods for skeleton sequence processing and recognition generally involve deforming the skeleton sequence by constructing suitable descriptors.
<cit.> used the star skeleton as a representative descriptor of the human skeleton, then converted the skeleton sequence into a symbol sequence that can be processed by the Hidden Markov Model. <cit.> proposed to represent skeleton sequences as curves in the Lie group, and the classification was performed by combining classifiers. 
<cit.> proposed using Hierarchical Gaussian Process Latent Variable Model to learn a hierarchical manifold space of motion patterns. 
With the development of research in recent years, the effectiveness of deep learning frameworks has been widely verified in various vision tasks <cit.>, which inspired people to leverage the discriminative feature extraction capabilities of those deep networks for skeleton modelling.
Du et al. <cit.> proposed a method that converts the skeleton sequence into an image format, which can be processed using CNNs for feature extraction. 
<cit.> further used colourisation to implicitly encoding the spatiotemporal information of the skeleton sequences. 
<cit.> generated multiple sets of clip-based representations and processed this information in parallel with the help of the Multi-Task Learning Network.
Inspired by the natural advantages of Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) in sequence modelling, methods such as <cit.> used RNNs or LSTMs to capture long-term contextual information.


GCN-based Methods.
As a typical message-passing network, the emergence of GCNs provides a new paradigm for skeleton sequence modelling.
Yan et al. <cit.> extended the basic GCNs operations to the spatiotemporal dimension. Shi et al. <cit.> moved away from the fixed graph modelling paradigm and proposed an adaptive relational representation. 
<cit.> proposed a complex disentangling and unifying operation based on the previous works, which significantly improved the model's classification performance. <cit.> constructed a more comprehensive graph learning framework by strengthening the representation granularity of channels.
Although the skeleton sequence contains relatively compact information and requires low computational overhead, researchers have recently conducted extensive research on recognition efficiency to avoid blindly stacking models.
Combined with high-level semantic information, Zhang et al. <cit.> proposed a simple yet effective semantics-guided neural network. 
<cit.> used efficient shift operations to implement the interaction of spatiotemporal information, significantly reducing the number of trainable parameters and computational complexity. 
Taking into account both accuracy and efficiency, Wu et al. <cit.> divided the channel dimensions and endowed them with diverse modelling capabilities, thereby improving the utilisation of network parameters.
Qin et al. <cit.> proposed a novel approach for constructing high-dimensional feature representations using angular encoding.
The study by Liu et al. <cit.> proposed a successful attack on existing skeletal graph models by exploiting spatiotemporal constraints.




 §.§ Graph Pooling Operation

In contrast to digital images, skeletons are typical non-Euclidean structural data with feature points that are not rigidly arranged and exhibit strong physical structure characteristics. As a result, pooling methods designed for image tasks, which typically involve selecting a regular rectangular area and obtaining a new feature point by averaging or taking the maximum value of all points in that area, are not directly applicable.
Instead, skeleton graph pooling often needs to consider both the graph's structure and the arrangement of its nodes. Specifically, it includes how to select the pooling area and how to generate a new feature representation.
There are three main approaches for graph pooling: hard rule, graph coarsening, and node selection.
Hard rule involves pre-setting pooling nodes and rules based on the known graph structure. 
Chen et al. <cit.> proposed a structure-based graph pooling scheme that considers the unique structure and prior topological knowledge of skeleton sequences.
Graph coarsening is a trainable version of the hard rule that involves clustering nodes and synthesising a super-node.
A differentiable graph pooling module constructed by <cit.> to map the nodes of each layer to a set of clusters. 
<cit.> introduced a graph-based clustering method for graph pooling using continuous relaxation of the normalised minimum segmentation problem and achieved cluster assignment by training Graph Neural Networks (GNNs). <cit.> proposed a hierarchical graph representation method using triplet pooling loss.
Node selection is to select some critical nodes to construct a new graph representation, thereby replacing the original structure. <cit.> proposed graph pooling and unpooling operations by selecting important nodes and restoring global nodes. <cit.> used the attention mechanism in GCNs to generate graph topology masks and used this mask to complete the pooling operation. 
In most previous studies, people often directly apply the graph pooling strategy to the skeleton task, failing to consider the structural properties of skeleton features and the specificity of different features.




§ THE PROPOSED APPROACH

In this section, we will begin by constructing the basic graph modelling module. Subsequently, the proposed innovations will be introduced step by step. Finally, we will describe the overall architecture that integrates all the components.



 §.§ Basic Graph Modelling Module


Referring to the standard graph definition, for any given graph G=(V, E), V represents the set of nodes, E represents the set of corresponding edges.
During the modelling process, each node aggregates information from its neighbouring nodes and updates its features accordingly.
Given the graph G_l at l_th layer, according to the defined rules, the 
newly generated graph can be expressed as,

    G_l+1=F(G_l,W_l) 
    	   =F^U((F^A(G_l,W_l^A)),W_l^U),

F represents the graph operation, W denotes the weight. The graph operation can be divided into two main components, namely aggregation F^A and updating F^U, where W^A and W^U represent the corresponding weights.

As described in <cit.>, spatial graph convolution involves performing a standard 2D convolution with a 1×1 kernel, followed by multiplying the resulting feature with a normalised adjacency matrix. 
Meanwhile, temporal graph convolution can be represented as a well-defined convolution operation on the constructed temporal graphs. 
In this work, we adopt existing methods as the backbone network to fairly measure the effectiveness of the proposed method.



 §.§ Structure Pooling Strategy with Region Awareness



In this section, we propose a novel structure pooling strategy (SSP) that incorporates region awareness to achieve more flexibility while preserving the underlying structure of the skeleton graph.
The graph convolution operation applied to skeleton-based action recognition typically involves both the feature itself and its corresponding relationship representation (usually represented by its adjacency relationship). 
Therefore, to perform pooling on the skeleton sequence, it is essential to not only select the pooling area but also take into account the transformation between the old and new graphs.
Additionally, it is necessary to generate a new relationship descriptor for the features that have undergone the pooling operation.



We begin by denoting the current features at t_th frame as x_t=[x_t0, x_t1,...,x_ti,...,x_tN], where x_ti∈𝐑^C corresponds to the feature of node v_i at frame t. We then divide the skeleton graph, which consists of N nodes, into M regions based on its physical structure, and then perform feature aggregation separately on each region to obtain a new graph with M nodes.
We generate the fundamental transformation matrix P∈ℝ^N× M between the old and new features. If joint i belongs to the j_th partitioned area (i.e., corresponds to the feature point j in the new graph), then P_ij=1; otherwise, P_ij=0.
While we believe that the above assignment setting is conducive in preserving structural characteristics, it ignores the specificity responses of each point across the entire graph or even across different input sequences.
For example, 'Rub Two Hands' is more focused on hand movement, while 'Jump Up' requires coordination of the entire body.

To tackle these challenges, we have imbued the pooling operation with the requisite flexibility without compromising its structural integrity. 
The entire calculation process is illustrated in Figure <ref> and Figure <ref>. 
Initially, we compute the correlation matrix R, which captures the correlation between each feature point and other locations.
The value associated with each feature point can be represented as the average of its connections with other feature points in the current graph, that is,

    ∀ i∈ N, R_ti=σ (∑_j=1^Nϕ(h(x_ti))^Tψ(h(x_tj))/N),

where σ(·) is the normalisation operation, and ϕ and ψ are projection functions that can be implemented by convolution filters, 
the dimension of project space is set as C/r. 
The entry R_ti measures the mean of the connection between point v_i and all other nodes at t_th frame, 
and this can be computed based on their corresponding latent representations h(·).
After that, the new feature after the spatial pooling operation can be expressed as,

    SPooling(x_t)   =x_tR_tP,

Residual connection is also introduced to achieve training stability.
That is, for each newly generated x^sp_tj,

    x^sp_tj=∑_i=1^Nx_ti (P_ij + R_ti P_ij).

Combing with (<ref>), we have

    x^sp_tj = ∑_i=1^Nx_ti (P_ij + σ(∑_j=1^Nϕ(h(x_ti))^Tψ(h(x_tj))/N) P_ij).

By introducing a trainable allocation matrix, our proposed strategies provide greater flexibility in treating each feature differently during the pooling operation. 
Furthermore, we retain the basic transformation form, which ensures the preservation of the structural characteristics of the skeleton sequence.
Finally, to obtain the relation representation for the newly pooled feature, we need to compute a new adjacency matrix that reflects the physical structure of the new graph. Since we maintain the basic structure in this pooling process, that is, the pooling is performed in a specific area, the feature representation and its physical structure associations can be conveniently obtained based on the pooling strategy. This enables the generation of the corresponding relationship description for each newly generated feature.



As for temporal pooling, due to the regular arrangement of the skeleton on the temporal dimension as discussed in Section <ref>, we can directly apply conventional pooling strategies.
The final pooled feature is obtained by performing an average pooling with a kernel of 2 on the current feature, as shown in  Figure <ref>, so the final formal can be written as,

    x^p   =STPooling(x)=TPooling(SPooling(x))  
       =TAP(x^sp)=(x^sp_[0::2]+x^sp_[1::2])/2.

TAP means the Average Pooling on the temporal dimension with a kernel size of 2, [0 :: 2] and [1 :: 2] represent to extract 1 frame every 2 frames starting from frame 0 and 1 respectively.




 §.§ Cross Fusion Block


The pooling operation significantly enhances the gathering and transmission of features and has the potential to reduce redundancy.
Bu tit may also have the opposite effect for skeletons due to
the irreparable and significant loss of effective information.
To alleviate the information loss during the pooling process, we maintain the basic pooling structure and construct a cross fusion block in a plug-and-play form.
As depicted in  Figure <ref>, the lower branch of the Cross Fusion Block represents the original graph pooling network module, while the newly introduced upper branch models obtain finer skeleton features that are cross-fused with the lower branch to enhance the representation ability of the original features.


As a fundamental component, we construct a graph pooling network module following the definitions in Section <ref> and Section <ref>; Specifically, we first obtain a new graph using the predefined pooling strategy and then perform a graph convolution operation on the newly generated feature representation. 
Specially,
for the current input feature of this block, based on the feature x_p obtained by pooling in Equation <ref>, the new representation can be expressed as,

    h=GraphConv(x^p),

where GraphConv general represents the graph convolution operation.
However, with this process, multiple nodes in the pooling area are replaced by a single new node representation before the graph operation. 
As a consequence, the subsequent graph convolution is executed on the compressed skeleton features with much lower dimension, resulting in the modelling of more abstract features. However, finer parts may not be fully captured in this process. Therefore, we need to emphasise the modelling of more refined features for our pooling network.
We first use graph convolution to further model the current features x to obtain a more informative feature representation x̂,

    x̂=GraphConv(x).

Then, the fusion of x̂ with h is performed to improve the overall representation ability. However, as x̂ and h have different resolutions, there is a misalignment issue between them.
To address this, we align the dimensions of these two features by performing a pooling operation on x̂ using h as a reference. Finally, the two aligned features are fused, as illustrated in  Figure <ref>.
According to the previous definition, we denote the upper branch feature after alignment as e,

    e=STPooling(x̂)=TPooling(SPooling(x̂))

this formula represents the alignment operation, where R(x̂) represents the relational representation calculated based on the current feature x̂, and the calculation process can be referred to Equation <ref>.
After that, the specific operations in the entire cross fusion block can be expressed as,

    y=s· h+(1-s)· e

where s means the corresponding weight, we will discuss it later in experimental section.



 §.§ Information Supplement Module


Given that the introduction of the pooling network significantly eliminates most information, it is reasonable to believe that the current network structure has not fully utilised its performance potential.
Rather than using conventional techniques such as distillation or pruning to compress the current network size, we opt to enhance the input features to fully exploit the modelling capabilities of the existing network structure.
Previous related methods often adopted complex multi-stream structures <cit.> to exploit the multi-dimensional representation information of input features. However, this strategy usually results in limited performance improvement while significantly increasing the computational and parameter requirements, thereby limiting the advantages of using skeleton features in real-world application deployment.

Compared to traditional methods, our approach involves performing appropriate pre-processing on the input side of the network to construct efficient input features while maintaining the feature extraction network, as illustrated in  Figure <ref>. We focus on two critical feature descriptors for the skeleton sequence: position-based and vector-based features, corresponding to joints and bones in the skeleton.
To align and integrate the position-based and vector-based features of the skeleton sequence, we use graph convolution to project them into a high-dimensional vector space. The projection operation maps each feature into a common vector space, enabling direct alignment and fusion. This results in a more informative representation that can capture both the joint and bone information.
These processes can be expressed as follows,

    f_p(x):xposition→x^pos,


    f_v(x):xvector→x^vec.

x^pos and x^vec correspond to the position-based and vector-based features obtained from the initial skeleton sequence, respectively, as mentioned in <cit.>.

    x^vec_in = GrpahConv(Norm(x^vec)),


    x^pos_in = GrpahConv(Norm(x^pos)).

The normalisation operation can be expressed as,

    Norm(x)=x-E(x)/√(var(x)+ϵ)*γ+β,

where γ and β are trainable parameters, E and var are the mean and standard-deviation.
The final input x_in is the fusion of position-based feature and vector-based feature.

    x_in = 
    Concat(x^vec_in,x^pos_in),

where Concat means feature concatenation along the channel dimension. 
The proposed input-strengthening strategy brings only a limited increase in the number of parameters and computational overhead compared to the original network structure but greatly enhances the representation of the new input features. The effectiveness of this strategy is verified through follow-up experiments.



 §.§ The Overall Structure

Combining the above descriptions, we construct the improved pooling network model starts with a baseline backbone taken from <cit.>, and replaces the basic graph convolution operation with the module proposed in <cit.>. We also make some optimisations to the data processing part following <cit.>. Our innovations are then added to the baseline, resulting in the improved pooling network model whose final structure is shown in  Figure <ref>. The improved pooling network model begins by processing the input feature using an information supplement module to obtain a new representation with richer information. The output of this module is then passed through several cross fusion blocks, where each block combines a graph pooling and a graph convolution operation. Finally, the classification results are obtained after global pooling and a classification layer.
We also remove the cross fusion block to construct a lightweight version, named IGPN-Light. For distinction, the default model is denoted as IGPN-Heavy.





§ EXPERIMENTS



 §.§ Datasets and Implementation Details
 

NTU-RGB+D. 
The NTU-RGB+D 60 dataset <cit.> comprises 56,880 videos captured from 80 camera views and 40 subjects, with 60 action categories. The NTU-RGB+D 120 dataset <cit.> extends the previous version to 120 action categories, with 114,480 videos captured from 155 views and 106 subjects.
To assess the effectiveness of our approach, we utilise the commonly adopted evaluation methods: cross-subject and cross-view for NTU-RGB+D 60,
cross-subject and cross-set for NTU-RGB+D 120.
For cross-subject evaluation, we use videos performed by half of the subjects for training and the remaining half for testing in both datasets. 
For cross-view evaluation, we use the samples collected by cameras with ID 2 and 3 for training and the remaining for testing in NTU-RGB+D 60. 
For cross-set evaluation, we use samples with even camera IDs for training and the others for testing in NTU-RGB+D 120.

UWA3D Multiview Activity II. 
The UWA3D Multiview Activity II Dataset <cit.> comprises 30 different classes performed by 10 characters, with each video captured by four different views (front, left, right, and top).
We randomly select two views as training data and test on the remaining two views separately. In our experiments, we report experimental results for all combinations of views, as well as the final average performance.





 §.§ Implementation Details

Training and evaluation. 
In general, our experiments are performed on the PyTorch platform, with cross-entropy loss and Stochastic Gradient Descent with Nesterov Momentum (0.9) employed as the training loss and optimisation algorithm, respectively. 
We use the processing steps from <cit.> for the NTU-RGB+D datasets. The initial learning rate is set to 0.1, with a batch size of 64 for NTU-RGB+D and 32 for UWA3D Multiview Activity II dataset. 
For NTU-RGB+D datasets, the learning rate is reduced to 1/10 of the previous epoch at the 35th and 55th epoch, and training stopped at the 65th epoch. 
For UWA3D Multiview Activity II dataset, the learning rate decay at 50th and 80th epochs, and the training stopped at the 100th epoch. 
We apply a warm-up strategy for the first five epochs across all datasets. 
Please find the details from Table <ref>.





Predefined spatial pooling rules. 
As shown in the Figure <ref>, we give the spatial skeleton graph in the whole network. The left side of the arrow is the spatial graph before pooling, and the right side of the arrow represents the newly generated graph after the structure spatial pooling operation.

As for the pooling rules, for the given skeleton representations, structured region division is performed first; with all nodes in each region condensed, a new sparse and compact graph representation can be obtained. 
Refer to <cit.>, the flow of several pooling operations is shown in the Table <ref>.












 §.§ A Comparison with the state-of-the-art

The effectiveness of the proposed method is verified through comprehensive comparisons with state-of-the-art methods on multiple benchmarks, as presented in Figure <ref>, Table <ref> and Table <ref>.

First, we conducted a performance comparison between IGPN and mainstream methods under single-stream evaluation.
As shown in Figure <ref>, the proposed IGPN outperforms HD-GCN <cit.>, InfoGCN <cit.>, CTR-GCN <cit.>, MS-G3D <cit.>, DC-GCN <cit.> in accuracy while maintain fewer Flops and parameters.
These results illustrate the efficiency and effectiveness of the proposed method, which destined its usability in real applications.


From Table <ref>, the results show demonstrate the proposed method is competitive with state-of-the-art methods. We observe that,  compared with the baseline method Graph2Net, both version of IGPN achieve significant improvement in accuracy on NTU-RGB+D 60 and NTU-RGB+D 120 datasets. 
CTR-GCN is one of the most representative methods published in recent years; 
compared with this method, IGPN-Heavy maintains a certain accuracy advantage. 
Following CTR-GCN, recent methods have shown limited improvements in accuracy. Compared with these methods, our method still achieves comparable performance.
On the UWA3D dataset, 
From Table <ref>, 
IGPN-Heavy outperforms previous methods in 7 out of 12 possible combinations, with a higher average performance when compared with previous state-of-the-art.

Our method has achieved a clear lead in both accuracy and efficiency under single-stream evaluation. Under multi-stream evaluation, our method has also achieved performance similar to or better than the optimal method in terms of accuracy.







 §.§ Ablation Studies

In this section, we present a comprehensive set of experiments and analyses for all of our proposed innovations on cross-subject of NTU-RGB+D 60.

Effects of different modules on performance.
To demonstrate the effectiveness of our innovations, we have chosen Graph2Net <cit.> with same number of layers as the baseline for comparison. As shown in Table <ref>, we conducted a thorough analysis of our proposed modules and found that the adaptive mechanism improved the model's accuracy to 89.1%. The fusion module further improved the accuracy by 0.8%. Finally, after supplementing the input features, the recognition performance reached 91.2%. The experimental results indicate a steady improvement in recognition accuracy by gradually adding our innovations. The final accuracy was significantly better than that of the original model and SGP.


Performance comparison on different backbones.
In this section, we investigate the applicability of our pooling strategy to other widely used methods, including AGCN <cit.>, Graph2Net <cit.> and CTR-GCN <cit.>. Specifically, we keep the basic framework unchanged and only replace specific graph convolution operations with these methods separately. The results in Table <ref> consistently demonstrate that the proposed improved pooling strategy can enhance model accuracy while reducing computational overhead significantly.
We choose the Graph2Net as the default graph convolution operation for all other experiments.



Analysis of specific configurations.
In this section, we will conduct detailed experiments and analyses on the parameter selection and structural design.






  * Structure pooling strategy with region awareness. 
As mentioned previously, we utilise SGP with the new implementation as the baseline to verify the impact of the normalisation function σ, the parameters of projection space r, as well as the location of the pooling operation. The experimental results are presented in Table <ref>.
Firstly, we observe that the recognition accuracy improves after incorporating the adaptive mechanism. Moreover, our model performs best when r=4 and Tanh is used as the normalisation function. 
Secondly, the model achieves an accuracy of 89.1% when the adaptive mechanism is introduced in all three pooling operations. 
Our also investigate the impact of retaining the initially assigned matrix through residual connection when introducing the adaptive mechanism. Results showed that removing the residual mechanism led to a drop in performance to 88.6%, highlighting the importance of preserving fundamental structural properties while introducing flexibility.



  * Cross fusion block. 
In our experiments, we investigated the impact of feature augmentation on performance when incorporating the new pooling strategy. Firstly, we explored the effect of the weight parameter in Equation <ref>. The results in Table <ref> show that the best performance is achieved when the weights of the two features being fused are both 0.5.
After fixing the weight s, we further investigated the optimal location and fusion strategy for the fusion module. The experimental results show that the performance improves as more fusion modules are added, ultimately reaching 89.9%. We also found that summation performs better than concatenation in the channel dimension. Notably, we perform the fusion after the global pooling operation for the last fusion.



  * Information supplement module. 
The experiments conducted in this section aimed to investigate the impact of parameter selection and structural design on the performance of the model. 
The results from Table <ref> show that batch normalisation on both position-based and vector-based features is necessary for better performance.
Also, changing the graph-based embedding operation to regular 2D convolution embedding resulted in a 0.5% decrease in performance. Furthermore, the optimal structure for graph embedding is found to be two layers of graph convolution with a final projected space dimension of 32.






Effects of different modules on performance.
We have presented the recognition accuracy of the models with the three proposed modules in each category, as shown in Figure <ref>. Based on the observations and analysis of the figure, several conclusions can be drawn:
The incremental incorporation of innovative modules has a discernible impact on the recognition accuracy of different categories.
For instance, the addition of the adaptive mechanism markedly enhances the recognition accuracy for behaviours such as drop and rub two hands, which necessitate more flexible pooling operations compared to the baseline. On the other hand, the fusion module performs exceptionally well in reading, wipe face, and other behaviours by providing more distinguishable multi-granularity feature representations.
Moreover, the results reveal that the introduction of these two innovations had a certain degree of adverse effects on some categories, such as putting on a shoe and taking a selfie. Fortunately, the supplementary modules has successfully mitigated these adverse effects.

Visualisation of region awareness.
To provide a more tangible illustration of our novel pooling method, we generated a visualisation of the region perception in  Figure <ref>, where the size of each point represents the importance of its corresponding position, reflecting the region that needs to be preserved during the pooling operation.
From (a), we can observe that the response at the hand joints is relatively large for the action 'rub two hands', and this trend is also evident in (b) and (c). For the action 'jump up' in (d) (e) (f), the response of the hands and legs is more evident due to the need for the cooperation of the entire body.
The observed phenomena demonstrate that our model can adaptively enhance information acquisition in critical regions for actions that focus on local areas or require whole-body coordination, providing discriminative feature expression during pooling.





Multi-stream fusion. 
The effectiveness of multi-stream network fusion has been extensively demonstrated in previous research <cit.>. We also adopt this approach to improve the classification performance of our algorithm further for a fairer comparison.
First, we separately verify the impact of spatial features and motion features on performance. The results in Table <ref> show that the recognition effect can be improved to 
91.6% and
92.3% by adopting the two-stream fusion strategy.
We also explored the use of an ensemble approach where we trained another separate network using only half of the frames. This technique is commonly used in other works such as <cit.>. We denoted this model as IGPN_En. The results in Table <ref> show that this ensemble mechanism improves the final accuracy to 
92.2% and
92.9% respectively.



Effects of different multi-stream strategies 
Since the designed ISM module includes joint-based and bone-based features, there may be confusion about the difference between this method and training joint-based and bone-based networks separately.
Indeed, mainstream methods<cit.> generally choose to train separate joint-based, bone-based, and corresponding motion-based features, and fuse the corresponding classification features to obtain the final recognition result of the multi-stream recognition network.
As shown in Table <ref>, we choose to delete the ISM module from the IGPN and train on different feature streams respectively, and finally fuse their results.
The results show that compared with our default method, this strategy not only brings a significant increase in computational effort (2.05GFlops vs 1.33GFlops on IGPN-Light,
5.38GFlops vs 3.83GFlops on IGPN-Heavy), and also leads to a decrease in accuracy performance
(92.0% vs 92.2% on IGPN-Light
92.5% vs 92.9% on IGPN-Heavy)
.



§ CONCLUSION

Existing skeleton-based graph pooling research suffers from the failure to consider graph properties of the skeleton, resulting in inflexibility and loss of vital information.
Here we have adopted several effective improvement strategies to enhance and supplement the pooling strategy from multiple perspectives.
(1) We comprehensively analysed the strengths and limitations of various traditional graph pooling strategies and devised a pooling strategy that significantly enhances flexibility while preserving the structural characteristics based on the local-global association.
(2) To address the issues of information loss and network under-saturation caused by pooling operations, we propose block-level multi-granularity representations and input-level discriminative information embedding.
On the above innovations, we constructed IGPN. The lightweight version can greatly improve efficiency and effect at the same time; the heavyweight version can further improve classification accuracy by sacrificing certain computing efficiency.

ACM-Reference-Format



