{'fraction_non_alphanumeric': 0.03410460351892022, 'fraction_numerical': 0.011689563750301278, 'mean_word_length': 4.848484848484849, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Object State Change Classification in Egocentric Videos using the Divided Space-Time Attention Mechanism

This report describes our submission called"TarHeels"for the Ego4D: Object State Change Classification Challenge. We use a transformer-based video recognition model and leverage the Divided Space-Time Attention mechanism for classifying object state change in egocentric videos. Our submission achieves the second-best performance in the challenge. Furthermore, we perform an ablation study to show that identifying object state change in egocentric videos requires temporal modeling ability. Lastly, we present several positive and negative examples to visualize our model's predictions. The code is publicly available at: https://github.com/md-mohaiminul/ObjectStateChange


Introduction
There has been rapid progress in the field of video classification in recent years [1-3, 6, 8, 9]. However, most existing datasets contain short videos from a third-person view. Moreover, many video recognition datasets are collected by a human in controlled environments. As a result, most of these datasets cannot be applied to many applications, for example, robotics and augmented reality, where we need to process videos from a first-person or egocentric point of view. Designing a video recognition model for egocentric videos can be difficult; because egocentric videos often contain many challenges such as camera movement, motion blur, object occlusion, etc.
In this challenge, we need to recognize how the camera wearer changes the state of an object by using or manipulating it in an egocentric video. One of the main challenges of this task is the generalization, i.e, the same object state change can be achieved in several ways. For example, a piece of wood can be cut into half in several ways, such as various tools, force, speed, grasps, etc., nevertheless, all should be recognized as the same object state change. For this challenge, we utilize a transformer-based video recognition model. First, we divide each frame of video into non-overlapping patches. Then we linearly project each patch into an embedding and augment it with positional information. Finally, we feed the sequence of patch-level embeddings into a transformer model. We leverage the divided space-time attention (TimeSformer [2]) mechanism for the transformer architecture, where temporal attention and spatial attention are applied separately within each block of the network. TimeSformer achieves state-of-the-art performance in several video classification benchmarks; however, most of these datasets contain videos of third-person view. In this work, we adopt TimeSformer for video recognition in egocentric videos, particularly for the object state change classification.
Our submission named "TarHeels" achieves the secondbest performance in the Ego4D: Object State Change Classification Challenge [7]. Moreover, we perform an ablation study with different attention mechanisms and show that identifying object state change in egocentric videos requires temporal modeling ability.

Our Approach
We follow the TimeSformer model proposed by [2]. Our model takes a video clip X ∈ R H×W ×3×F as input, where H is the height, W is the width, and F is the number of frames. Then we divide each frame into N non-overlapping patches of size P × P , where N = HW/P 2 . This is equivalent to applying a convolution with equal kernel size and stride of P × P . Then a linear layer is applied to project each patch to a latent dimension of D, and a positional embedding E ∈ R N ×D is added to each patch embedding.
The resulting sequence of patch embedding is passed through a transformer encoder which is a stack of L transformer blocks. Each transformer block contains a multi-headed attention (MHA) and a multi-layer perceptron (MLP) block. Layer normalization (LN) is applied before each block, and a skip connection layer is added after each block.
We follow the divided space-time attention schemes, which decomposes the self-attention operation over the temporal dimension and spatial dimension. First temporal attention is computed, where the self-attention weights α (l,a) (p,t) of later l and attention head a for a patch at spatial location p and temporal location t is computed by the following equation.
(1) Here, SM denotes the softmax activation, and D h is the dimensionality of the attention heard, where, D h = D/A, A being the number of attention heads. Similarly, spatial attention is computed by the following equation.
(2) We also experiment with two other attention schemes, i.e, Space-Only attention and Joint Space-Time Attention. In the Space-Only attention mechanism, attention weights are computed only considering spatial tokens within the same frame. Whereas, in the Joint Space-Time attention scheme, attention weights are calculated considering all spatial and temporal tokens of the video clip. Note that, computing attention over one dimension (e.g., spatial or temporal) significantly reduces the computational cost. In Figure 1, we visualize Space-Only, Joint Space-Time, and Divided Space-Time attention blocks.

Implementation Details
Model Architecture. We follow TimeSformer-base architecture, which operates on a video of dimension 8 × 224 × 224. We divide each video frame using a patch size of 16 × 16. We use ViT-base [5] as the transformer backbone, which has 12 self-attention layers with 12 heads on each layer. It uses a hidden dimension 768, and MLP dimension 3072. The transformer backbone is pre-trained on the Imagenet-21k dataset [4]. Training. We resize each video to a height and width of 256 × 320 and use a sampling rate of 3/2 fps. During training, we sample 8 consecutive frames from a video and randomly sample a crop of the size of 224 × 224 from each frame. Then we feed the 8 frames video clip to our transformer model. We train our model for 15 epochs with an initial learning rate of 0.005 and divide the learning rate by 10 at epochs 11, and 14. We train our model using 8 NVIDIA RTX A6000 GPUs using a batch size of 16. We Inference. During inference, we randomly sample 3 clips of 8 frames from a video. For each temporal clip, we construct 3 clips by randomly cropping video frames to dimensions 224 × 224. Thus, we construct 9 clips for each video, and the final prediction is obtained by averaging the softmax scores of these 9 predictions.

Experimental Results
Leader Board Results. In Table 1, we present leaderboard results on the unseen test set of Ego4D: Object State Change Classification Challenge [7]. The performance is evaluated by classification accuracy. Our challenge entry named "TarHeels" achieves second-best performance among all challenge entries Ablation Studies.
We ablate different self-attention schemes and analyze the importance of the temporal modeling for the Object State Change Classification. Particularly, we train our model 5 epochs using three selfattention mechanisms (Space-only, Joint Space-Time, Divided Space-Time) and present the performance of the validation set in Table 2 cates the importance of temporal modeling for identifying object state change. Secondly, the divided Space-Time attention mechanism leads to the best accuracy showing the effectiveness of this method for this particular task. Figure 2 shows several videos of the validation set of the Object State Change Classification task with their ground truth labels and our predictions. Specifically, we present one true positive, one false positive, one true negative, and one false negative example. Here, we show video frames at 1 fps.

Conclusion
In this report, we describe our submission ("TarHeels") in the Ego4D: Object State Change Classification Challenge. Leveraging the Divided Space-Time attention, we achieve the second-best result on this task. In the future, we will explore more about the egocentric aspects of the videos, and try to improve our model based on the intrinsic properties of egocentric videos. We would also investigate architectures that utilize inductive bias of visual domains, for example, multi-scale video models. Figure 2. We present several videos of the validation set of the Object State Change Classification task with their ground truth labels and our predictions. Here, we show video frames at 1 fps.