Action Localization through Continual Predictive Learning
Sathyanarayanan N. Aakur Sudeep Sarkar
cs.CV




headings
2129  

 












Action Localization through Continual Predictive Learning





SN. Aakur et al.



Oklahoma State University, Stllwater, OK, 74074 

saakurn@okstate.edu University of South Florida, Tampa, FL 33620 

sarkar@usf.edu




Action Localization through Continual Predictive Learning
    Sathyanarayanan N. Aakur^1 Sudeep Sarkar^2
    December 30, 2023
=========================================================





The problem of action recognition involves locating the action in the video, both over time and spatially in the image. The dominant current approaches use supervised learning to solve this problem, and require large amounts of annotated training data, in the form of frame-level bounding box annotations around the region of interest. In this paper, we present a new approach based on continual learning that uses feature-level predictions for self-supervision. It does not require any training annotations in terms of frame-level bounding boxes. The approach is inspired by cognitive models of visual event perception that propose a prediction-based approach to event understanding. We use a stack of LSTMs coupled with CNN encoder, along with novel attention mechanisms, to model the events in the video and use this model to predict high-level features for the future frames. The prediction errors are used to continuously learn the parameters of the models.  This self-supervised framework is not complicated as other approaches but is very effective in learning robust visual representations for both labeling and localization. It should be noted that the approach outputs in a streaming fashion, requiring only a single pass through the video, making it amenable for real-time processing. We demonstrate this on three datasets - UCF Sports, JHMDB, and THUMOS'13 and show that the proposed approach outperforms weakly-supervised and unsupervised baselines and obtains competitive performance compared to fully supervised baselines. Finally, we show that the proposed framework can generalize to egocentric videos and obtain state-of-the-art results in unsupervised gaze prediction. 







§ INTRODUCTION

We develop a framework for jointly learning spatial and temporal localization through continual, self-supervised learning, in a streaming fashion, requiring only a single pass through the video. 
Visual understanding tasks in computer vision have focused on the problem of recognition <cit.> and captioning <cit.>, with the underlying assumption that each input video is already localized both spatially and temporally. 
While there has been tremendous progress in action localization, it has primarily been driven by the dependence on large amounts of tedious, spatial-temporal annotations. 
In this work, we aim to tackle the problem of spatial-temporal segmentation of streaming videos in a continual, self-supervised manner, without any training annotations.



Drawing inspiration from psychology <cit.>, we consider the underlying mechanism for both event understanding and attention selection in humans as the idea of predictability. 
Defined as the surprise-attention hypothesis <cit.>, unpredictable factors such as large changes in motion, appearance, or goals of the actor have a substantial effect on the event perception and human attention. 
Longer-term, temporal surprise has shown to have a strong correlation with event boundary detection <cit.>, whereas short-term spatial surprise (such as those caused by motion) have a stronger effect on human attention and localization <cit.>.
Our approach combines both spatial and temporal surprise to formulate a computational framework to tackle the problem of self-supervised action localization in streaming videos in a continual manner. 

We formulate our computational framework on the idea of spatial-temporal feature anticipation for modeling predictability of perceptual features. The main assumption in our framework is that expected, unpredictable features require attention and often point to the actor performing the action of interest, whereas predictable features can belong to background clutter not relevant to the action of interest. It is to be noted that unpredictability or surprise is not the same as rarity. It refers to short-term changes that aid in the completion of an overall task, which can be recurring. We model the perceptual features using a hierarchical, cyclical, and recurrent framework, whose predictions are influenced by current and prior observations as well as current perceptual predictions. Hence, the predictive model’s output can influence the perception of the current frame being observed. The predictions are constantly compared with the incoming observations to provide self-supervision to guide future predictions. 

We leverage these characteristics to derive and quantify spatial-temporal predictability. Our framework performs continuous, prediction-based learning to generate “attention maps” that are consistent with the predictability of the observed scene. With the attention derived from spatial-temporal unpredictability, we leverage advances in region proposals <cit.> to localize actions in streaming videos without any supervision. Contrary to other attention-based approaches <cit.>, we do not use the object-level characteristics such as label, role, and affordance in the proposal generation process. 

Contributions: The contributions of our approach are three-fold: (i) we are among the first to tackle the problem of self-supervised action localization in streaming videos without any training data such as labels or bounding boxes, (ii) we show that modeling spatial-temporal prediction error can yield consistent localization performance across action classes and (iii) we show that the approach generalizes to egocentric videos and achieves competitive performance on the unsupervised gaze prediction task. 



§ RELATED WORK

Supervised action localization approaches tackle the action localization problem through the simultaneous generation of bounding box proposals and labeling each bounding box with the predicted action class. Both the bounding box generation and labeling are fully supervised, i.e., they require ground truth annotations of both bounding boxes and labels. The typical approach is to leverage advances in object detection to include temporal information <cit.> for proposal generation. The final step typically involves the use of the Viterbi algorithm <cit.> to link the generated bounding boxes across time.

Weakly-supervised action localization approaches <cit.> have been explored to reduce the need for extensive annotations. They typically only require video-level labels and rely on object detection-based approaches to generate bounding box proposals. It is to be noted that weakly supervised approaches also use object-level labels and characteristics to guide the bounding box selection process. Some approaches <cit.> use a similarity-based tracker to connect bounding boxes across time to incorporate temporal consistency into the generation process.

Unsupervised action localization approaches have not been explored to the same extent as supervised and weakly-supervised approaches. These approaches do not require any supervision - both labels or bounding boxes. The two more common approaches are to generate action proposals using (i) super-voxels <cit.> and (ii) clustering motion trajectories <cit.>. It should be noted that  <cit.> also uses object characteristics to evaluate the “humanness” of each super-voxel to select bounding box proposals. 
Our approach falls into the class of unsupervised action localization approaches. The most closely related approaches (with respect to architecture and theme) to ours are VideoLSTM <cit.> and Actor Supervision <cit.>, which use attention in the selection process for generating bounding box proposals, but require video-level labels. 
We, on the other hand, do not require any labels or bounding box annotations for training. 

While fully supervised approaches return more precise localization and achieve better recognition, the required number of annotations is rather large and is not amenable to an increase in the number of classes and a decrease in the number of training videos. Weakly supervised approaches, while not requiring frame-level annotations, have the underlying assumption that there exists a large, annotated training set that allows for effective detection of all possible actors (both human and non-human) in the set of action classes. Unsupervised approaches, such as ours, do not make any such assumptions but can result in poorer localization performance. We alleviate this to an extent by leveraging advances in region proposal mechanisms and learning robust representations for obtaining video-level labels. 




§ SELF-SUPERVISED ACTION LOCALIZATION

In this section, we introduce our self-supervised action localization framework, as illustrated in Figure <ref>. Our approach has four core components: (i) feature extraction and spatial region proposal, (ii) a self-supervised future prediction framework, (iii) a spatial-temporal error detection module, and (iv) the error-based action localization process. 


 §.§ Feature Extraction and Spatial Region Proposal

The first step in our approach is feature extraction and the subsequent per-frame region proposal generation for identifying possible areas of actions and associated objects. Considering the tremendous advances in deep learning architectures for learning robust spatial representations, we use pre-trained convolutional neural networks to extract the spatial features for each frame in the video.
We use a region proposal module, based on these spatial features, to predict possible action-agnostic spatial locations. 
We use class-agnostic proposals (i.e., the object category is ignored, and only feature-based localizations are taken into account) at this stage for two primary reasons. First, we do not want to make any assumptions on the actor's characteristics, such as label, role, and affordance. Second, despite significant progress in object detection, there can be a lot of missed detections, especially when the object (or actor) performs actions that can transform their physical appearance. 
It is to be noted that these considerations can result in a large number of region proposals which require careful and robust selection but can yield higher chances of correct localization. 



 §.§ Self-supervised Future Prediction

The second stage in our proposed framework is the self-supervised future prediction framework. 
We consider the future prediction module to be a generative model whose output is conditioned on two factors - the current observation and an internal event model. The current observation f^S_t is the feature-level encoding of the presently observed frame I_t. We use the same feature encoder as the region proposal module to reduce the memory footprint and complexity of the approach. 
The internal event model is a set of parameters that can effectively capture the spatial-temporal dynamics of the observed event. Formally, we define the predictor model as P(f̂^S_t+1|W_e, f^S_t), where W_e represents the internal event model and f̂^S_t+1 is the predicted features at time t+1. Note that features f^S_t is not a one-dimensional vector, but a tensor (of dimension w_f × h_f × d_f) representing the features at each spatial location. 

We model temporal dynamics of the observed event using Long Short Term Memory Networks (LSTMs)<cit.>. While other approaches <cit.> can be used for prediction, we consider LSTMs to be more suited for the following reasons. First, we want to model the temporal dynamics across all frames of the observed action (or event). Second, LSTMs can allow for multiple possible futures and hence will not tend to average the outcomes of these possible futures, as can be the case with other prediction models. Third, since we work with error-based localization, using LSTMs can ensure that the learning process propagates the spatial-temporal error across time and can yield progressively better predictions, especially for actions of longer duration.

Formally, we can express LSTMs as 



    i_t = σ(W_ix_t + W_hih_t-1 + b_i) 
     
    f_t = σ(W_fx_t + W_hfh_t-1 + b_f ) 
     
    o_t = σ(W_ox_t + W_hoh_t-1 + b_o ) 
     
    g_t = ϕ(W_gx_t + W_hgh_t-1 + b_g ) 
    
    m_t = f_t · m_t-1 + i_t · g_t 
    
    h_t = o_t ·ϕ(m_t)


where x_t is the input at time t, σ is a non-linear activation function, (·) represents element-wise multiplication, ϕ is the hyperbolic tangent function (tanh) and W_k and b_k represent the trained weights and biases for each of the gates.

As opposed to <cit.>, who also use an LSTM-based predictor and a decoder network, we use a hierarchical LSTM model (with three LSTM layers) as our event model. This modification allows us to model both spatial and temporal dependencies, since each higher-level LSTMs act as a progressive decoder framework that captures the temporal dependencies captured by the lower-level LSTMs. The first LSTM captures the spatial dependency that is propagated up the prediction stack. 
The updated hidden state of the first (bottom) LSTM layer (h^1_t) depends on the current observation f^S_t, the previous hidden state (h^1_t-1) and memory state (m^1_t-1).
Each of the higher-level LSTMs at level l take the output of the bottom LSTM's output h^l-1_t and memory state m^l-1_t and can be defined as (h^l_t, m^l_t) = LSTM(h^l_t-1, h^l-1_t, m^l-1_t). Note this is different from a typical hierarchical LSTM model <cit.> in that the higher LSTMs are impacted by the output of the lower level LSTMs at current time step, as opposed to that from the previous time step. Collectively, the event model W_e is described by the learnable parameters and their respective biases from the hierarchical LSTM stack.

Hence, the top layer of the prediction stack acts as the decoder whose goal is to predict the next feature f^S_t+1 given all previous predictions f̂^S_1, f̂^S_2, …f̂^S_t, an event model W_e and the current observation f^S_t. We model this prediction function as a log-linear model characterized by


    log p(f̂^s_t+1 | h^l_t) = 
    ∑_n=1^t f(W_e, f^S_t) + log Z(h_t)


where h^l_t is the hidden state of the l^th level LSTM at time t and Z(h_t) is a normalization constant. The LSTM prediction stack acts as a generative process for anticipating future features.

The objective function for training the predictive stack is a weighted zero order hold between the predicted features and the actual observed features, weighted by the zero order hold difference. The prediction error at time t is given by

    E(t) = 1/n_f∑^w_f_i=1∑^h_f_j=1m̂_t(i,j)⊙‖ f^S_t+1(i,j) - f̂^S_t+1(i,j)‖_ℓ_1^2

where each feature f^S_t has dimensions w_f × h_f × d_f and m̂_t(i,j) is a function that returns the zero order difference between the observed features at times t and t+1 at location (i,j). Note that the prediction is done at the feature level and not at the pixel level, which would result in errors at a different granularity. 



 §.§ Prediction Error-based Attention Map

At the core of our approach is the idea of spatial-temporal prediction error for localizing the actions of interest in the video. It takes into account the quality of the predictions made and the relative alignment of the prediction errors in and around each spatial location. 
The input to the error detection module is the unaveraged result of the objective function from Equation <ref>, which by itself represents the combined spatial-temporal loss. 
We compute a weight α_ij associated with each spatial location (i,j) in the predicted feature f̂^S_t+1 as

    α_ij = exp(e_ij)/∑_m=1^w_k∑_n=1^h_kexp(e_mn)

where e_ij represents the weighted prediction error at (i,j) as defined in Equation <ref> and can be considered to be a function a of the state of the top-most LSTM and the input feature f^S_t at time t and can be defined as a(f^S_t, h^l_t-1). The resulting matrix is an error-based attention map that allows us to localize the prediction error at a specific spatial location where as Equation <ref> allows for temporal localization. 
It is to be noted that this is very similar in formulation of Bahdanau attention <cit.>. However, there are two key differences. First, our formulation is not parametrized and does not add to the number of learnable parameters in the framework. Second, our attention map is a characterization of the difficulty in anticipating unpredictable motion whereas Bahdanau attention is an effort to increase the encoding ability of the decoder and does not characterize the unpredictability of the future feature. We compare the use of both types of attention in Section <ref>, where we see that error-based localization is more suitable for our application.



 §.§ Extraction of Action Tubes

The action localization module receives a stream of bounding box proposals and an error-based attention map to select an output tube, parametrized as a selection of bounding boxes 𝒯 = ℬ_it at each time instance t. 
The action localization is a selection algorithm that filters all region proposals from Section <ref> and returns the collection of proposals that have a higher probability of action localization. We do so by assigning an energy term to each of the bounding box proposals (ℬ_it) at time t and choosing the top k bounding boxes with least energy as our final proposals. The energy of a bounding box ℬ_it is defined as 

    E(ℬ_it) = w_α ϕ(α_ij, ℬ_it) + w_t δ(ℬ_it, {ℬ_j,t-1})

where ϕ(·) is a function that returns a value characteristic of the distance between the bounding box center and location of maximum error, δ(·) is a function that returns the minimum spatial distance between the current bounding box and the closest bounding box from the previous time step and w_α and w_t are scaling factors. Note that δ(·) is introduced to enforce temporal consistency in predictions, but we find that it is optional since the LSTM prediction stack implicitly enforces the temporal consistency through its memory states. In our experiments we set k=10, w_α = 0.75.



 §.§ Implementation Details

In our experiments, we use a VGG-16 <cit.> network pre-trained on ImageNet as our feature extraction network. We use the output of the last convolutional layer before the fully connected layers as our spatial features. Hence the dimensions of the spatial features are w_f=14, h_f=14, d_f=512. These output features were then used by an SSD <cit.> to generate bounding box proposals. Note that we just take the generated bounding box proposals without taking into account classes and associated probabilities. 
We use a three layer hierarchical LSTM model with the hidden state size as 512 as our predictor module. We use the vanilla LSTM as proposed in  <cit.>. 
Video level-features are obtained by max-pooling the element-wise dot-product of hidden state of the top-most LSTM and the attention values across time.
We train with the adaptive learning mechanism proposed in  <cit.>, with the initial learning rate set to be 1×10^-8 and scaling factors Δ^-_t and Δ^+_t as 1×10^-2 and 1×10^-3, respectively. The network was trained for 1 epoch on a computer with one Titan X Pascal. 




§ EXPERIMENTAL SETUP



 §.§ Data

We evaluate our approach on three publicly available datasets for evaluating the proposed approach on the action localization task. 

UCF Sports <cit.> is an action localization dataset consisting of 10  classes of sports actions such as skating and lifting collected from sports broadcasts. It is an interesting dataset since it has a high concentration of distinct scenes and motions that make it challenging for localization and recognition. We use the splits (103 training and 47 testing videos) as defined in  <cit.> for evaluation.

JHMDB <cit.> is composed of 21 action classes and 928 trimmed videos. All videos are annotated with human-joints for every frame. The ground truth bounding box for the action localization task is chosen such that the box encompasses all the joints.  This dataset offers several challenges, such as increasing amounts of background clutter, high inter-class similarity, complex motion (including camera motion), and occluded objects of interest. We report all results as the average across all three splits.

THUMOS'13 <cit.> is a subset of the UCF-101 <cit.> dataset, consisting of 24 classes and 3,207 videos. Ground truth bounding boxes are provided for each of the classes for the action localization task. It is also known as the UCF-101-24 dataset. Following prior works <cit.>, we perform our experiments and report results on the first split.

We also evaluate the generalization ability of the proposed approach on egocentric videos by evaluating on the unsupervised gaze prediction task. There has been evidence from cognitive psychology that there is a strong correlation between gaze points and action localization <cit.>. Hence, the gaze prediction task would be a reasonable measure of the generalization to action localization in egocentric videos. We evaluate the performance on the GTEA Gaze <cit.> dataset, which consists of 17 sequences of tasks performed by 14 subjects, with each sequence lasting about 4 minutes. We use the official splits for the GTEA datasets as defined in prior works  <cit.>. 



 §.§ Metrics and Baselines

For the action localization task, we follow prior works <cit.> and report the mean average precision (mAP) at various overlap thresholds, obtained by computing the Intersection Over Union (IoU) of the predicted and ground truth bounding boxes. We also evaluate the quality of bounding box proposals by measuring the average, per-frame IoU, and the bounding box recall at varying overlap ratios. 

Since ours is an unsupervised approach, we obtain class labels by clustering the learned representations using the k-means algorithm. While more complicated clustering may yield better recognition results <cit.>, the k-means approach allows us to evaluate the robustness of learned features. We evaluate our approach in two settings K_gt and K_opt, where the number of clusters is set to the number of ground truth action classes and an optimal number obtained through the elbow method <cit.>, respectively. From our experiments, we observe that K_opt is typically three times the number of ground truth classes, which is not unreasonable and has been a working assumption in other deep learning-based clustering approaches <cit.>.  

We also compare against other LSTM and attention-based approaches (Section <ref>) to the action localization problem for evaluating the effectiveness of the proposed training protocol.

For the gaze prediction task, we evaluate the approaches using Area Under the Curve (AUC), which measures the area under the curve on saliency maps for true positive versus false-positive rates under various threshold values. We also report the Average Angular Error (AAE), which measures the angular distance between the predicted and ground truth gaze positions. 
Since the output of our model is a saliency map, AUC is a more appropriate metric compared to average angular error (AAE), which requires specific locations.




§ QUANTITATIVE EVALUATION

In this section, we present the quantitative evaluation of our approach on two different tasks, namely action localization, and egocentric gaze prediction. For the action localization task, we evaluate our approach on two aspects - the quality of proposals and spatial-temporal localization.



 §.§ Quality of Localization Proposals

We first evaluate the quality of our localization proposals by assuming perfect class prediction. This allows us to independently assess the quality of localization performed in a self-supervised manner. We present the results of the evaluation in Table <ref> and compare against fully supervised, weakly supervised, and unsupervised baselines. As can be seen, we outperform many supervised and weakly supervised baselines. APT <cit.> achieves a higher localization score. However, it produces, on average, 1,500 proposals per video, whereas our approach returns approximately 10 proposals. A large number of localization proposals per video can lead to higher recall and IoU but makes the localization task i.e., action labeling per video harder and can affect the ability to generalize across domains. 

Also, it should be noted that our approach produces proposals in streaming fashion, as opposed to many of the other approaches, which produce action tubes based on motion computed across the entire video. This can make real-time action localization in streaming videos harder. 




 §.§ Spatial-temporal Action Localization

We also evaluate our approach on the spatial-temporal localization task. This evaluation allows us to analyze the robustness of the self-supervised features learned through prediction. 
We generate video-level class labels through clustering and use the standard evaluation metrics (Section  <ref>) to quantify the performance. 
The AUC curves with respect to varying overlap thresholds are presented in Figure <ref>. We compare against a mix of supervised, weakly-supervised, and unsupervised baselines on all three datasets. 

On the UCF Sports dataset (Figure <ref>(a)), we outperform all baselines including several supervised baselines except for Gkioxari and Malik <cit.> at higher overlap thresholds (σ > 0.4) when we set number of clusters k to the number of ground truth classes. When we allow for some over-segmentation and use the optimal number of clusters, we outperform all baselines till σ > 0.5.



On the JHMDB dataset (Figure <ref>(b)), we find that our approach, while having high recall (77.8% @ σ=0.5), the large camera motion and intra-class variations have a significant impact on the classification accuracy. Hence, the mAP suffers when we set k to be the number of ground truth classes. When we set the number of clusters to the optimal number of clusters, we outperform other baselines at lower thresholds (σ<0.5). It should be noted that the other unsupervised baseline (Soomro et al <cit.>) uses object detection proposals from a Faster R-CNN backbone to score the "humanness" of a proposal. This assumption tends to make the approach biased towards human-centered action localization and affects its ability to generalize towards actions with non-human actors. We, on the other hand, do not make any assumptions on the characteristics of the actor, scene, or any motion dynamics. 

On the THUMOS'13 dataset (Figure <ref>(c)), we achieve consistent improvements over unsupervised and weakly supervised baselines, at k=k_gt and achieve state-of-the-art mAP scores when k=k_opt. It is interesting to note that we perform competitively (when k=k_gt) the weakly-supervised attention-based VideoLSTM <cit.>, which uses a convLSTM for temporal modeling along with a CNN-based spatial attention mechanism. It should be noted that we have a higher recall rate (0.47@σ=0.4 and 0.33@σ=0.5)  at higher thresholds compared to other state-of-the-art approaches on THUMOS'13 and shows the robustness of the error-based localization approach to intra-class variation and occlusion. 

Clustering quality. Since there is a significant difference in the mAP score when we set a different number of clusters in k-means, we measured the homogeneity (or purity) of the clustering. The homogeneity score measures the “quality” of the cluster by measuring how well a cluster models a given ground-truth class. Since we allow the over-segmentation of clusters when we set k to the optimal number of clusters, this is an important measure of feature robustness. Higher homogeneity indicates that intra-class variations are captured since all data points in a given cluster belong to the same ground truth class. We observe an average homogeneity score of 74.56% when k is set to the number of ground truth classes and 78.97% when we use the optimal number of clusters. As can be seen, although we over-segment, each of the clusters typically models a single action class to a high degree of integrity. 



 §.§ Comparison with other LSTM-based approaches

We also compare our approach with other LSTM-based and attention-based models to highlight the importance of the proposed self-supervised learning paradigm. 
Since LSTM-based frameworks can have highly similar architectures, we take into account different requirements and characteristics, such as the level of annotations required for training and the number of localization proposals returned per video. 
We compare with three approaches similar in spirit to our approach - ALSTM <cit.>, VideoLSTM <cit.> and  Actor Supervision <cit.> and summarize the results in Table <ref>. It can be seen that we significantly outperform VideoLSTM and ALSTM on the THUMOS'13 dataset in both recall and mAP@σ=0.2. 



Actor Supervision <cit.> outperforms our approach on recall, but it is to be noted that the region proposals are dependent on two factors - (i) object detection-based actor proposals and (ii) a filtering mechanism that limits proposals based on ground truth action classes, which can increase the training requirements and limit generalizability. Also, note that returning a higher number of localization proposals can increase recall at the cost of generalization. 



 §.§ Ablative Studies

The proposed approach has three major units that affect its performance the most - (i) the region proposal module, (ii) future prediction module, and (iii) error-based action localization module. In our primary framework, we use the prediction error-based localization to choose bounding box proposals generated through a class-agnostic SSD object detection model. We make the SSD class-agnostic by only considering bounding box proposals at all detection thresholds. 
We consider and evaluate several alternatives to all three modules. We choose selective search <cit.> and EdgeBox <cit.> as alternative region proposal methods. 
We use an attention-based localization method for action localization as an approximation of the ALSTM <cit.> to evaluate the effectiveness of using the proposed error-based localization. 

We also evaluate the effect of attention-based prediction by introducing a Bahdanau <cit.> attention layer before prediction. 
We evaluate these approaches on the UCF Sports dataset and visualize the results in Figure <ref>(c). It can be seen that the use of the prediction error-based localization has a significant improvement over a trained attention-based localization approach. We can also see that the choice of region proposal methods do have some effect on the performance of the approach, with selective search and EdgeBox proposals doing slightly better at higher thresholds (σ∈ (0.4, 0.5)) at the cost of inference time and additional bounding box proposals (50 compared to the 10 from SSD-based region proposal). Using SSD for generating proposals allows us to share weights across the frame encoder and region proposal tasks and hence reduce the memory and computational footprint of the approach. 
We also find that using attention as part of the prediction module significantly impacts the performance of the architecture. It could, arguably, be attributed to the objective function, which aims to minimize the prediction error. Using attention to encode the input could impact the prediction function. 



 §.§ Unsupervised Egocentric Gaze Prediction

Finally, we evaluate the ability to generalize to egocentric videos by quantifying the performance of the model on the unsupervised gaze prediction task. Given that we do not need any annotations or other auxiliary data, we employ the same architecture and training strategy for this task. We evaluate on the GTEA gaze dataset and compare it with other unsupervised models in Table <ref>. 


As can be seen, we obtain competitive results on the gaze prediction task, outperforming all baselines on both the AUC and AAE scores. It is to be noted that we outperform the center bias method on the AUC metric. Center bias exploits the spatial bias in egocentric images and always predicts the center of the video frame as the predicted gaze position. The significant improvement in the AUC metric indicates that our approach predicts gaze fixations that are more closely aligned with the ground truth than the center bias approach. Although we do not return a specific gaze position, we outperform all baselines except center bias on the AAE metric. Given that the model was not designed explicitly for this task, it is a remarkable performance, especially given the performance of fully supervised baselines such as DFG <cit.>, which achieve 10.6 and 88.3 for AUC and AAE.




 §.§ Qualitative Evaluation

We also qualitatively analyze the proposed approach to identify its strengths and weaknesses. We find that our approach has a consistently high recall for the localization task across datasets and domains. We consider that an action is correctly localized if the average IoU across all frames is higher than 0.5, which indicates that most, if not all, frames in a video are correctly localized. 



We illustrate the recall scores and subsequent AUC scores for each class in the UCF sports dataset in Figure <ref>(a-b). It can be seen that for many classes (7/10 to be specific), we have more than 80% recall at an overlap threshold of 0.5. We find, through visual inspection, that the spatial-temporal error is often correlated with the actor, but is often not at the center of the region of interest and thus reduces the quality of the chosen proposals. We illustrate this effect in Figure <ref>. The first row shows the input frame, the second shows the error-based attention, and the last row shows the final localization proposals. If more proposals are returned (as is the case with selective search and EdgeBox), we can obtain a higher recall (Figure <ref>(b)) and higher mAP. More qualitative results in the supplementary.



§ CONCLUSION

In this work, we introduce a self-supervised approach to action localization, driven by spatial-temporal error localization. We show that the use of self-supervised prediction using video frames can help learn highly robust features and obtain state-of-the-art results on localization without any training annotations. We also show that the proposed framework can work with a variety of proposal generation methods 
without losing performance. We also show that the approach can generalize to egocentric videos without changing the training methodology or the framework and obtain competitive performance on the unsupervised gaze prediction task.





splncs04

