Learning Discriminative Spatio-temporal Representations for Semi-supervised Action Recognition
Yu Wang Sanping Zhou Kun Xia Le Wang





































  IAIR, Xi'an Jiaotong University
  
  
  Xi'an, Shaanxi
  China
  

wy05140234@stu.xjtu.edu.cn



  IAIR, Xi'an Jiaotong University
  
  Xi'an, Shaanxi
  
  China
spzhou@xjtu.edu.cn



  IAIR, Xi'an Jiaotong University
  
  Xi'an, Shaanxi
  
  China
xiakun@stu.xjtu.edu.cn



  IAIR, Xi'an Jiaotong University
  
  Xi'an, Shaanxi
  
  China
lewang@xjtu.edu.cn










































































    Semi-supervised action recognition aims to improve spatio-temporal reasoning ability with a few labeled data in conjunction with a large amount of unlabeled data.
    
    Albeit recent advancements, existing powerful methods are still prone to making ambiguous predictions under scarce labeled data, embodied as the limitation of distinguishing different actions with similar spatio-temporal information. 
    In this paper, we approach this problem by empowering the model two aspects of capability, namely discriminative spatial modeling and temporal structure modeling for learning discriminative spatio-temporal representations.
    Specifically, we propose an Adaptive Contrastive Learning (ACL) strategy. It assesses the confidence of all unlabeled samples by the class prototypes of the labeled data, and adaptively selects positive-negative samples from a pseudo-labeled sample bank to construct contrastive learning.
    Additionally, we introduce a Multi-scale Temporal Learning (MTL) strategy. It could highlight informative semantics from long-term clips and integrate them into the short-term clip while suppressing noisy information.
    
    Afterwards, both of these two new techniques are integrated in a unified framework to encourage the model to make accurate predictions. Extensive experiments on UCF101, HMDB51 and Kinetics400 show the superiority of our method over prior state-of-the-art approaches.

   







<ccs2012>
   <concept>
       <concept_id>10010147.10010178.10010224.10010225.10010228</concept_id>
       <concept_desc>Computing methodologies Activity recognition and understanding</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
 </ccs2012>


[500]Computing methodologies Activity recognition and understanding

























Learning Discriminative Spatio-temporal Representations for Semi-supervised Action Recognition
    Le Wang
    
==============================================================================================






§ INTRODUCTION



Action recognition is one of the most basic topics in video understanding, which has been widely applied to many real-world scenarios, including human-computer interaction <cit.>, autonomous driving <cit.>, and so on <cit.>. However, most existing methods <cit.> heavily rely on large-scale and well-annotated training data, which is very tedious and time-consuming.
Therefore, semi-supervised action recognition has attracted growing attentions in academia and industry, which only requires a few labeled data along with a large amount of unlabeled data.

 




Existing performant methods have achieved remarkable successes in the past few years.
MvPL <cit.> leverages more indicative information from different views by introducing additional temporal gradients and optical flow modalities.
CMPL <cit.> introduces auxiliary networks to obtain more pseudo labels for efficient semi-supervised learning. 


LTG <cit.> improves the representation of the RGB data branch with the help of the temporal gradient modality. 
TimeBalance <cit.> proposes to learn temporally-invariant and temporally-distinctive features to improve the video representation.

In contrast, SVFormer <cit.> achieves new state-of-the-art performance arguably due to its powerful capability in modeling inter-frame relationships.




Albeit achieving notable advances, they are still prone to making ambiguous predictions under only scarce labeled data, limiting the model's ability to distinguishing different actions with similar spatio-temporal information.

Two illustrative examples from the powerful method SVFormer <cit.> are depicted in Figure <ref>. 
An action of “Nunchucks” is mistakenly recognized as “Tai Chi” due to similar spatial representations and backgrounds between them. Besides, the model is prone to make biased predictions for an action of “High Jump” since it shares similar sub-actions and temporal structures with “Long Jump”, hindering the model’s performance.

Building upon the above observations, we propose a new paradigm for semi-supervised action recognition by emphasizing learning discriminative spatio-temporal representations. This paradigm involves an Adaptive Contrastive Learning (ACL) strategy and a Multi-scale Temporal Learning (MTL) strategy, both of which encourage the model to make accurate predictions.

Concretely, ACL aims at improving the model's ability of discriminative spatial modeling. 
During training iteration process, it dynamically constructs class prototypes by using labeled samples and updates a momentum memory bank that stores pseudo-labeled samples.
We compute the distance between each pseudo-labeled sample and its corresponding class prototype and encode the distance as its reliability score by a two-component Gaussian Mixture Model (GMM).
For an unlabeled sample with higher reliability score, we construct contrastive learning by selecting samples of the same category with a reliability score above the threshold from the memory bank as positive samples and the rest as negatives. If it has lower reliability score, we take the whole memory bank as negative samples and its weakly-augmented sample as the positive sample for contrastive learning.



Thus, the proposed ACL empowers the model to learn the discriminative spatial representations between semantic actions.



Additionally, we present a MTL strategy to equip the model with temporal structure modeling capability. Specifically, MTL first obtains long-term clips of different scales via different sampling intervals on unlabeled videos in addition to the original short-term clip. 
Next, we design a cross-scale temporal calibration module, which  could highlight informative semantics from long-term clips and integrate them into the short-term clip while suppressing noisy information. 
Finally, we align the representations from the calibrated long-term clips and the short-term clip, enabling the model to learn the temporal differences between different scales of clips.


The main contributions of this paper can be summarized as follows:

    
  * This paper approaches semi-supervised action recognition from a new perspective by learning discriminative spatio-temporal representations. It encourages the model to distinguish different actions with similar spatio-temporal information under scarce labeled data.
    
  * We propose an Adaptive Contrastive Learning (ACL) strategy and a Multi-scale Temporal Learning (MTL) strategy to empower the model two aspects of capability, i.e., discriminative spatial modeling and temporal structure modeling.
    
    
  * We integrate ACL and MTL into a unified framework, which significantly advances state-of-the-art results. Extensive experiments on the UCF101, HMDB51 and Kinetics400 datasets demonstrate the effectiveness of the proposed framework.






§ RELATED WORK

Semi-supervised Learning In Image Classification.
Recently, semi-supervised learning has been widely explored in the field of image classification. Previous studies predominantly adopt consistency regularization <cit.> or pseudo-labeling <cit.>. Consistency regularization encourages the model to have similar outputs for similar inputs, which requires the model to be robust to perturbations. The pseudo-labeling method requires the model to assign pseudo-labels to unlabeled images based on predictions. Some of the recent
works  <cit.> combine consistency regularization and pseudo-labeling, requiring strong augmented predictions to match weakly augmented ones. In addition, another set of methods  <cit.> demonstrates the effectiveness of self-supervision in this field. However, these methods do not model the temporal structure of human actions and do not consider discriminative  spatial representations between different actions, resulting in poor performance in the field of semi-supervised action recognition.


Semi-supervised Learning In Action Recognition.
Recently, significant progress has been made in the field of semi-supervised action recognition. 
MvPL <cit.> and LTG <cit.> achieve higher quality pseudo-labels by introducing additional modalities, such as optical flow, temporal gradients, etc. TCL <cit.> explores the impact of group contrast loss. CMPL <cit.> introduces an auxiliary network during training, providing complementary gains for unlabeled videos. Timebalance <cit.> learns complementary temporal information through various forms of self-supervised learning. SVFormer <cit.> demonstrates the superiority of transformer architecture in the semi-supervised action recognition field and proposes corresponding  data augmentation methods. Compared to these methods, our approach focuses on learning the discriminative spatio-temporal representations, which does not rely on additional modalities and outperforms the state of the art methods on multiple public benchmarks.


Contrastive Learning In Image and Video.
The essence of contrastive learning lies in maximizing the similarity between positive sample pairs and encouraging discrimination among negative samples. In recent years, significant progress has been made in the field of image contrastive learning <cit.>. SimCLR <cit.> trains a single encoder network with a large batch size to ensure sufficient positive and negative samples for learning. 
BYOL <cit.> introduces a prediction head concept, employing an asymmetric architecture for contrastive learning without negative samples. 

In addition, notable advancements occurs in video contrastive learning <cit.>. 
Compared to image contrastive learning, video contrastive learning also explores enabling models to learn rich temporal variations through contrastive learning. Some methods <cit.> model the positions of frames, tending to learn inter-frame relationships. 

Other methods <cit.> focus more on the temporal structure of actions, enabling models to understand the relationship between global actions and local sub-actions. What's different, we propose a new strategy for exploring temporal structure modeling in semi-supervised action recognition. At the same time, we integrate labeled data to assess the confidence of unlabeled samples and adaptively select positive and negative samples for spatial contrastive learning.



§ METHOD

This paper introduces a unified framework for semi-supervised action recognition, emphasizing learning discriminative spatio-temporal representations to enable the model to make accurate predictions. An overview of the proposed framework is illustrated in Figure <ref>. Next, we will elaborate our method step by step.


 §.§ Problem Setting

Semi-supervised action recognition aims to learn a decent model for accurate action recognition from a smaller labeled dataset X_L = {( x_i^l,y_i) }_i = 1^N_l and a larger unlabeled dataset X_U = { x_i^u}_i = 1^N_u, where N_l and N_u are the numbers of labeled videos and unlabeled videos.
For a labeled sample x_i^l, its ground truth label is an action category y_i. The major challenge of semi-supervised action recognition is to improve spatio-temporal reasoning ability especially with scarce labeled data.







 §.§ Basic Framework



We adopt the teacher-student network <cit.> as our basic framework, in which the teacher is implemented as an Exponential Moving Average (EMA) of the student.




For training on all labeled data X_L, the student network is optimized through the standard cross-entropy loss:


    L_l = - 1/N_l∑_i = 1^N_ly_ilogℱ_θ_s( A_w( x_i^l) ),

where ℱ_θ_s represents the student network and A_w(·) represents the weak augmentation. 
For training on unlabeled data X_U, we first apply different intensities of data augmentation to each unlabeled data x_i^u <cit.>. Then, we use the teacher model ℱ_θ_t to generate corresponding pseudo labels ŷ_i.

Then, the student model ℱ_θ_s is trained on unlabeled data with pseudo-labels using the cross-entropy loss function:

    L_u = - 1/N_u∑_i = 1^N_u𝕀( max( 
    ℱ_θ_t( A_w( x_i^u)) ) > δ)ŷ_ilogℱ_θ_s( A_s( x_i^u) ),

where A_s(·) represents the strong augmentation. δ is a predefined fixed threshold. 𝕀 denotes an indicator function, which equals 1 when the maximum category probability exceeds δ, and 0 otherwise. It is used to select high-quality pseudo-labels. So, we define the total loss function as:

    L_base = L_l + α L_u,

where α controls the contribution of the unsupervised cross-entropy loss.





 §.§ Adaptive Contrastive Learning


We present the proposed adaptive contrastive learning strategy in Figure <ref>, which aims to enhance the spatial modeling capability of the model. In this module, we first use labeled data X_L to construct class prototypes. For ∀ x^l∈ X_L, we could obtain its feature vector f^l:

    f^l = h_s( f_θ_s( A_w( x^l) ) ).

where f_θ_s( · ) and h_s( · ) represent the encoder and spatial projection head of the student network, respectively. Then, we iteratively update the class prototypes by calculating the exponential moving average of f^l:

    X_(t) = ( 1 - β) ·f^l + β·X_(t-1),

where X_(t) and X_(t-1) are the prototype of an arbitrary class at the t-th iteration and (t-1)-th iteration respectively. β is empirically set to 0.9. 


For ∀ x^u∈ X_U, we first apply strong augmentation A_s(·) and weak augmentation A_w(·) to it. Then, we feed them separately into the student and the teacher encoder with spatial projection heads to generate the anchor sample f^a and its naive positive sample f^p:

    {f^a = h_s( f_θ_s( A_s( x^u) ) )
    f^p = h_t( f_θ_t( A_w( x^u) ) ).,

where f_θ_t( · ) and h_t( · ) represent the encoder and spatial projection head of the teacher network respectively. The parameters of h_t( · ) are the exponential moving average of h_s( · ). In order to ensure the diversity of spatial features, we establish a momentum memory bank 𝕄 <cit.> that dynamically stores each unlabeled sample f with its pseudo label ŷ from the teacher network during model training. 


Key aspect of efficient contrastive learning lies in selecting high-quality positive and negative samples. An intuitive idea is to choose the samples of the same category from 𝕄 as reliable positives. 

Given a training sample x^u with its pseudo label ŷ^c, we could construct an initial positive sample set ℕ̂^+:



    ℕ̂^+ = {f| ŷ = ŷ^c,f∈𝕄}∪{f^p}.



However, pseudo labels may be noisy and unreliable. Thus, we further evaluate their confidences based on the class prototypes. 

We first compute the cosine distances between the samples from ℕ̂^+ and the corresponding class prototypes.

For the i-th sample f_i in ℕ̂^+, the cosine distance could be formulated as:

    dis( f_i,X^ŷ^c) = f_i·X^ŷ^c/∥f_i∥_2·∥X^ŷ^c∥_2,

where ∥·∥_2 is the L_2 norm. Afterwards, we input all dis( f_i,X^ŷ^c) into a two-component Gaussian Mixture Model (GMM) to generate the reliability score γ_i for each f_i:

    γ_i = GMM( dis( f_i,X^ŷ^c) | { dis( f_i,X^ŷ^c) }_i = 1^| ℕ̂^+|),

where γ_i∈ 0,1. To select reliable positive samples from ℕ̂^+ based on reliability scores, we set a threshold ε and consider those with reliability scores greater than ε as the final positive samples, denoted as 

    ℕ^+ = {f_i| γ_i > ε,f_i∈ℕ̂^+}.


Therefore, we could construct contrastive learning for x^u by selecting positive samples from ℕ^+ and negative samples from 𝕄-ℕ^+, respectively. However, the reliability score of x^u may be very low due to the limited performance of the model under scarce labeled data, resulting in its pseudo label error. So, we construct contrastive learning through taking the whole memory bank 𝕄 as the negative samples and only f^p as the positive sample when its γ≤ε.


Finally, the loss of adaptive contrastive learning is expressed in the following formula:

    L_ACL = - log. ∑_f∈ℕ^+exp( f.^a·f/τ)/. ∑_f∈ℕ^+exp( f.^a·f/τ). + ∑_f∈𝕄-ℕ^+exp( f.^a·f/τ)

where τ is the temperature hyperparameter. During the training process, we select different positive and negative samples based on the reliability score of x_u. The summary of ACL is presented in Algorithm <ref>.




 §.§ Multi-scale Temporal Learning



As in the previous analysis, the model is prone to make ambiguous predictions for different actions that share similar temporal structures, especially when few labeled data is available. To this end, we propose a Multi-scale Temporal Learning (MTL) strategy to improve the ability of model's temporal structure modeling, as shown in Figure <ref>.


Given an unlabeled video x^u ∈ X_U, we randomly sample a short-term clip and multiple long-term clips by using different sampling rates. All clips include T frames and we represent the short-term clip as x^S and the long-term clips as x^L_n, where n = 1, 2, …, N. N is a hyperparameter that controls the number of long-term clips and N=2 in the experiments.


Afterwards, we enter them into the teacher network and the student network respectively to obtain the query and key:

    {f^q =  f_θ_s( A_w( x^S) )
    f^k_n =  f_θ_t( A_w( x^L_n) )..

Then, we design a cross-scale temporal calibration module, aiming to suppress the parts in each scale of long-term clips that are unrelated to the short-term clip. We calculate the similarity between f^q and f^k_n as follows:

    A_n = f^q·f^k_n/∥f^q∥_2·∥f^k_n∥_2,

where ∥·∥ denotes the L_2-norm. A_n is the generated attention map, where semantic positions unrelated to the short-term clip are assigned low weights. Then, we calibrate long-term clips by multiplying A_n with f^k_n:

    f^k^*_n = A_n·f^k_n,

where f^k^*_n represents the representation of the calibrated long-term clip. We also apply strong enhancement to the short-term clip and encode it as feature f^q^*:

    f^q^* = f_θ_s( A_s( x^S) ).


Then, we feed both of them to their respective temporal projection heads to obtain the final representations:

    {z^q_n = g_s_n( f^q^*)
    z^k_n = g_t_n( f^k_n^*).,

where g_s_n( · ) and g_t_n( · ) respectively represent the temporal projection heads of the student network and the teacher network.
The parameters of g_t_n( · ) are also the exponential moving average of g_s_n( · ). 
Then, following DINO <cit.>, we use softmax function with temperature hyperparameters τ_s and τ_t to normalize the outputs z^q_n and z^k_n. The output probability distributions P^q_n and P^k_n are used to calculate alignment loss as follows:


    L_n = H( P^k_n,P^q_n),

where H( a,b) = - alogb. The final MTL loss is as follows:

    L_MTL = 1/N∑_n = 1^NL_n.






 §.§ Training Objective


During the training stage, we make some adjustments to L_base in Section <ref>. Firstly, we obtain the pseudo-label for each unlabeled data x_i^u by taking the average of the predictions of multiple clips in Section <ref>:

    ŷ_i = argmax( ℱ_θ_t( A_w( x^S)) + ∑_n = 1^Nℱ_θ_t( A_w( x^L_n) )).

Then, we use the reliability score γ_i obtained in Section <ref> to replace the fixed weight α. Finally, the complete loss function is as follows:

    L = L_base + μ_1· L_MTL + μ_2· L_ACL,

where μ_1 and μ_2 are responsible for controlling the contributions of L_MTL and L_ACL.







§ EXPERIMENT




In this section, we introduce the experimental settings in Section <ref> and Section <ref>. From Section <ref> to Section <ref>, we conduct experiments under different labeling rates with ablation experiments and empirical analysis. We also show the visualization in Section <ref>. Note that we only use the RGB modality and the official validation set for inference.



 §.§ Datasets and Evaluation

Datasets.

Following the previous performant semi-supervised action recognition methods <cit.>, we evaluate our method on three public action recognition benchmarks: UCF-101 <cit.>, HMDB-51 <cit.>, and Kinetics400 <cit.>. 

UCF-101 is a widely used video dataset, which includes 13,220 videos belonging to 101 classes.
We randomly select 1 or 10 samples from each class as labeled sets following SVFormer <cit.>. 

HMDB-51 is a smaller dataset with 6,776 videos across 51 classes. We follow the splits of VideoSSL <cit.> and LTG <cit.> and conduct experiments at three different labeling rates: 40%, 50%, 60%. 

Kinetics-400 is a large-scale dataset consisting of approximately 245k training videos and 20k validation videos belonging to 400 classes. For Kinetics-400, we follow SVFormer <cit.> and TimeBalance <cit.>, randomly sampling 6 and 60 videos for each class under 1% and 10% settings, forming two balanced labeled subsets.

Evaluation Metric.

We report Top-1 accuracy for major comparisons and Top-5 accuracy in some ablation studies.



 §.§ Implementation Details


Baseline.
We follow SVFormer <cit.> combined with TimeSformer <cit.> as our baseline, which is also initialized with weights from ImageNet <cit.>. 
Compared with the traditional convolutional neural network (CNN) based methods, SVFormer could achieve significant performance gains by only requiring the same number of learnable parameters.



Training Details.
For the training process, we follow the settings of TimeSformer <cit.>. All experiments are done with the SGD optimizer during training, with a momentum of 0.9 and weight decay of 0.001. We set the initial learning rate as 0.005 and divide it by 10 at the 25-th epoch and the 28-th epoch. 
The weak augmentation is implemented through random scaling and random cropping while the strong augmentation works by adding additional grayscale and color jitter to the weakly-augmented samples.

In addition, we set the confidence threshold δ to 0.3 and set temperature hyperparametersτ, τ_s, and τ_t are set to 0.07, 0.1, and 0.04 respectively. The loss weights μ_1 and μ_1 are both set to 1. 
According to <cit.>, B_l and B_u are set to 1 and 5 respectively. Each clip consists of 8 frames. ε is set to 0.7, and it will be further analyzed in subsequent ablation experiments. The sampling steps for short-term clips are 8, and 16 and 32 for long-term clips.

Inference. During the testing phase, we follow the recent state-of-the-art method <cit.>, sampling five segments from the entire video and create three different crops to achieve a resolution of 224×224. The final prediction is the average of the softmax probabilities of these 5×3 predictions.





 §.§ Comparison with State-of-the-art Methods


In order to demonstrate the capability of our proposed method, we compare it with the latest semi-supervised action recognition methods on three public datasets, including UCF101 <cit.>, HMDB51 <cit.>, and Kinetics400 <cit.>. 



Comparison with CNN-based methods.

We report the comparison results with 13 CNN-based methods in Table <ref>. In general, our method does not use additional modalities and significantly outperforms all CNN-based methods by a large margin on all labeling ratios.














































Comparison with Transformer-based methods.
Compared with the transformer-based method <cit.>, our proposed method records the new state-of-the-art performance on the three benchmarks. Specifically, our method outperforms it by an average of 7.9% on UCF101 dataset and 2.1% on HMDB51 dataset at the same inference cost.

This further confirms the importance and superiority of enhancing  discriminative spatial modeling and temporal structure modeling.









 §.§ Ablation Study


To better understand how the proposed method works, we conduct a series of ablation studies on UCF-101 at the 1% labeling ratio setting and HMDB51 at the 40% labeling ratio setting. Our method follows the same semi-supervised learning pattern as FixMatch <cit.> and MeanTeacher <cit.>.




Contributions of different training components.

To validate our key designs, we ablate the effects of the proposed adaptive contrastive learning (ACL) and multi-scale temporal learning (MTL). 


For this purpose, we established four experimental configurations, as shown in Table <ref>: (1) baseline, which corresponds to the basic framework mentioned in Section <ref> and has been adjusted according to Section <ref> to validate the effect of enhancing only discriminative spatio-temporal representations. (2) only ACL, (3) only MTL, (4) both. It can be observed that the baseline has achieved decent results but still significantly lower than the other three configurations. This indicates that relying solely on pseudo-labels for semi-supervised action recognition is insufficient with limited labeled data. Furthermore, applying our ACL strategy significantly improves performance on both datasets, which suggests that discriminative spatial features play a crucial role for more accurate predictions. Additionally, equipping the baseline model with the MTL strategy alone also lead to significant performance gains, indicating that modeling temporal structures contributes to enhancing the model's discriminative capacity. Finally, the both strategies improve the spatio-temporal reasoning ability of the model in a cooperative way, as achieving both on top of the baseline outperforms either one alone.





























































Analysis of Adaptive Contrastive Learning. 
Regarding our ACL module, an intuitive question is how the performance would be if we do not consider the confidence of features. Therefore, we compare the proposed ACL with two alternative contrastive learning based methods, MoCo <cit.> and SimCLR <cit.>. 

We conduct an ablation study on the UCF-101 dataset with 1% labeling ratio and use same data augmentations and experiment setting for fair comparison.
As shown in Table <ref>, it can be observed that all methods improve the performance, proving the effectiveness of spatial representation modeling. In addition, we observe that ACL is better than MoCo and SimCLR. It is reasonable because ACL considers sample confidence and reduces noise in contrastive learning by adaptively selecting positive and negative samples, which demonstrates the necessity of considering feature confidence in spatial contrastive learning for semi-supervised action recognition.


Analysis of threshold ε. 
The threshold ε is used to evaluate the reliability of samples in ACL. The larger value of ε means that the fewer samples are assigned as positive samples in the momentum memory bank while the lower ε may take some unreliable samples with noisy pseudo labels as positive samples, degenerating the model performance. 

To this end, we conduct an ablation study on the choice of hyper-parameter ε. From Table <ref>, we can observe that the performance peak around ε = 0.7. In addition, we ablate the case where the confidence of the unlabeled sample x^u is not considered. We can see that the performance will decrease due to the introduction of noise.


Analysis of Multi-scale Temporal Learning.
Learning from multi-scale temporal information could enable the model to learn the temporal structures of different semantic actions.
We conduct an experiment on how different scales of temporal information impacts the model performance. 

As shown in Table <ref>, we set up four experimental settings: (1) none, (2) only long-term x^L_1 (3) only long-term x^L_2 and (4) multi-scale temporal information. The experimental results indicate that temporal information at different scales contributes to learning discriminative temporal structures, 
which improves the temporal modeling ability of the model to distinguish different actions with similar sub-actions.




 §.§ Visualization


In Figure <ref>, we show the visualization of embeddings with and without enhancing discriminative spatio-temporal representation. Generally, due to limited labeled samples in semi-supervised learning, it is difficult to for models to form accurate clusters. We observe that after applying our spatio-temporal representation enhancement, the embeddings of the same categories cluster better together. In addition, we also show the model's predictions for the same blurred  sample, as shown in Figure <ref>. It can be observed that, after enhancing discriminative spatio-temporal representations, the model exhibits greater discriminability for actions with similar spatio-temporal information.







 §.§ Performance gains


In Table <ref>, we observe that our framework achieves an average gain of more than 30% for categories prone to ambiguous predictions Javelin Throw, Soccer Juggling, High Jump, Juggling Balls and YoYo. ACL enhances the model's discriminative spatial modeling capabilities and more accurately predicts actions YoYo and Juggling Balls that rely on spatial representation. For actions like Javelin Throw and High Jump that rely on temporal context and temporal structure, MTL enables the model to learn temporal information at different scales to enhance the ability to distinguish these actions.































§ CONCLUSION


In this paper, we propose a new semi-supervised action recognition learning framework, which achieves more accurate predictions by enhancing discriminative spatio-temporal representations. Our method introduces adaptive contrastive learning and multi-scale temporal learning strategies, and integrates them into a unified framework. Our approach significantly surpasses all previous methods and achieves state-of-the-art performance on the UCF-101, HMDB-51, and Kinetics-400 datasets with different label ratios. In the future, we plan to investigate the effectiveness of learning discriminative representations in other video-based tasks and explore new methods for learning discriminative representations.

ACM-Reference-Format



