Spatio-Temporal Dynamics and Semantic Attribute Enriched Visual Encoding for Video Captioning
Nayyer Aafaq Naveed Akhtar Wei Liu Syed Zulqarnain Gilani Ajmal Mian









Spatio-Temporal Dynamics and Semantic Attribute Enriched
 Visual Encoding for Video Captioning
    Nayyer Aafaq  Naveed Akhtar   Wei Liu  Syed Zulqarnain Gilani  Ajmal Mian

Computer Science and Software Engineering,

The University of Western Australia.

nayyer.aafaq@research.uwa.edu.au, 
{naveed.akhtar, wei.liu, syed.gilani, ajmal.mian}@uwa.edu.au

    December 30, 2023
=================================================================================================================================================================================================================================================================





  Automatic generation of video captions is a fundamental challenge in computer vision. Recent techniques typically employ a combination of Convolutional Neural Networks (CNNs) and Recursive Neural Networks (RNNs) for video captioning. These methods mainly focus on tailoring sequence learning through RNNs for better caption generation, whereas off-the-shelf visual features are borrowed from CNNs. We argue that careful designing of visual features for this task is equally important, and present a visual feature encoding technique to generate semantically rich captions using Gated Recurrent Units (GRUs). Our method embeds rich temporal dynamics in visual features by hierarchically applying  Short Fourier Transform to CNN features of the whole video. It additionally derives high level semantics from an object detector to enrich the representation with spatial dynamics of the detected objects. The final representation is projected to a compact space and fed to a language model. By learning a relatively simple language model comprising two GRU layers, we establish new state-of-the-art on MSVD  and MSR-VTT datasets for METEOR and ROUGE_L metrics.  







§ INTRODUCTION


Describing videos in natural language is  trivial for humans, however it is a very complex task for machines. To generate meaningful video captions, machines are required to understand objects, their interaction, spatio-temporal order of events and other such minutiae in videos; yet, have the ability to articulate these details in grammatically correct and meaningful natural language sentences <cit.>. 
The bicephalic nature of this problem has recently led researchers from Computer Vision and Natural Language Processing (NLP) to combine efforts in addressing its challenges <cit.>.
Incidentally, wide applications of video captioning in emerging technologies, e.g. procedure generation from instructional videos <cit.>, video indexing and retrieval <cit.>; have recently caused it to  receive attention as a fundamental task in Computer Vision. 

Early methods in video captioning and description, e.g. <cit.> primarily aimed at generating the correct Subject, Verb and Object (a.k.a. SVO-Triplet) in the captions. More recent methods <cit.> rely on Deep Learning <cit.> to build frameworks resembling a typical neural machine translation system that can generate a single sentence <cit.> or multiple sentences <cit.>  to describe  videos. 
The two-pronged problem of video captioning provides a default division for the deep learning  methods to encode visual contents of videos using Convolutional Neural Networks (CNNs) <cit.> and decode those into captions using language models. Recurrent Neural Networks (RNNs) <cit.> are the natural choice for the latter component of the problem.   

Since semantically correct sentence generation has a longer history in the field of NLP, deep learning based captioning techniques mainly focus on language modelling <cit.>.  
For visual encoding, these methods forward pass video frames through a pre-trained 2D CNN; or a video clip through a 3D CNN, and extract features from an inner layer of the network - referred as  `extraction layer'. 
Features of frames/clips are commonly combined with mean pooling to compute the final representation of the whole video.
This, and similar other visual encoding techniques <cit.>  - due to the nascency of video captioning research - grossly under-exploit the prowess of visual representation for the captioning task. To the best of our knowledge, this paper presents the first work that concentrates on improving the visual encoding mechanism for the captioning task.     



We propose a visual encoding technique to compute representations enriched with spatio-temporal dynamics of the scene, while also accounting for the high-level semantic attributes of the videos.
Our visual code (`v' in Fig. <ref>) fuses information from multiple sources. We process activations of 2D and 3D CNN extraction layers by hierarchically applying Short Fourier Transform <cit.> to them, where  InceptionResNetv2 <cit.> and C3D <cit.> are used as the 2D and 3D CNNs respectively.  The proposed neuron-wise activation transformation using whole videos results in encoding fine temporal dynamics of the scenes.   
We encode spatial dynamics by processing objects' locations and their multiplicity information extracted from an Object Detector (YOLO <cit.>). The semantics attached to the output layers of the Object Detector and the 3D CNN are also exploited to embed high-level semantic attributes in our visual codes.
We compress the visual codes and learn a language model using the resulting representation. With highly rich visual codes, a relatively simple Gated Recurrent Unit (GRU) network is proposed for language modeling, comprising two layers, that already results in on-par or better performance compared to the existing sophisticated models <cit.> on multiple evaluation metrics.  

The main contributions of this paper are as follows. We propose a visual encoding technique that effectively encapsulates spatio-temporal dynamics of the videos and embeds relevant high-level semantic attributes in the visual codes for video captioning. The proposed visual features contain the detected object attributes, their frequency of occurrences as well as the evolution of their locations over time. We establish the effectiveness of the proposed encoding by learning a GRU-based language model and perform thorough experimentation on  MSVD <cit.> and MSR-VTT <cit.> datasets. Our method  achieves up to 2.64% and 2.44% gain in the state-of-the-art on METEOR and ROUGE_L metrics for these datasets.  







§ RELATED WORK

Classical methods in video captioning commonly use template based techniques in  which Subject (S), Verb (V), and Object (O)  are detected separately and then joined together in a sentence.  

However, the advancement  of deep learning research has also transcended to modern video captioning methods. The latest approaches in this direction generally exploit deep learning for visual feature encoding as well as its decoding into meaningful captions. 

In template based approaches, the first successful video captioning method was proposed by Kojima et al. <cit.> that focuses  on describing videos of one person performing one action only. Their heavy reliance on the correctness of manually created activity concept hierarchy and state transition model prevented its extension to more complex videos. Hanckmann et al. <cit.> proposed a method to automatically describe events involving multiple actions (seven on average), performed by one or more individuals. Whilst most of the prior work was restricted to constrained domains <cit.>, Krishnamoorthy et al. <cit.> led the early works of describing open domain videos. <cit.> proposed semantic hierarchies to establish relationships between actor, action and objects. <cit.> used CRF to model the relationship between visual entities and treated video description as a machine translation problem. However, the aforementioned approaches depend on predefined sentence templates and fill in the template by detecting entities from classical methods. These approaches are not sufficient for the syntactically rich sentence generation to describe open domain videos. 

In contrast to the methods mentioned above, deep models directly generate sentences given a visual input. For example LSTM-YT <cit.> feed in visual contents of video obtained by average pooling all the frames into LSTM and produce the sentences. LSTM-E <cit.> explores the relevance between the visual context and sentence semantics. The initial visual features in this framework were obtained using 2D-CNN and 3D-CNN whereas the final video representation was achieved by average pooling the features from frames / clips neglecting the temporal dynamics of the video. TA <cit.> explored the temporal domain of video by introducing an attention mechanism to assign weights to the features of each frame and later fused them based on attention weights. S2VT <cit.> incorporated optical flow to cater for the temporal information of the video. SCN-LSTM <cit.> proposed semantic compositional network that can detect the semantic concepts from mean pooled visual content of the video and fed that information into a language model to generate captions with more relevant words. LSTM-TSA<cit.> proposed a transfer unit that extracts semantic attributes from both images as well as mean pooled visual content of videos and added it as a complementary information to the video representation to further improve the quality of caption generation. M^3-VC <cit.> proposed a multi-model memory network to cater for long term visual-textual dependency and to guide the visual attention.

Even though the above methods have employed deep learning, they have used mean pooled visual features or attention based high level features from CNNs. These features have been used directly in their framework in the language model or by introducing additional unit in the standard framework. We argue that this technique under-utilizes the state of the art CNN features in video captioning framework. We propose features that are rich in visual content and empirically show that this enrichment of visual features alone when combined with a standard and simple language model can outperform existing state of the art methods. Visual features are part of every video captioning framework. Hence, instead of using high level or mean pooled features, building on top of our visual features can further enhance the video captioning frameworks' performances. 



§ PROPOSED APPROACH

Let 𝒱 denote a video that has `f' frames or `c' clips. The fundamental task in automatic video captioning is to generate a  textual sentence 𝒮 = {𝒲_1, 𝒲_2,..., 𝒲_w} comprising `w' words that matches closely to human generated captions for the same video. Deep learning based video captioning methods typically define an energy loss function of the following form for this task:

    Ξ ( v, 𝒮) = - ∑_t = 1^w  logPr( 𝒲_t |  v, 𝒲_1,...𝒲_t-1),






where Pr(.) denotes the probability, and v∈ℝ^d is a visual representation of 𝒱.
By minimizing the cost defined as the  Expected value of the energy Ξ(.) over a large corpus of videos, it is hoped that the inferred model ℳ can automatically generate meaningful captions for  unseen videos.


In this formulation, `v' is considered a training input, that makes remainder of the problem a sequence learning task.

Consequently, the existing methods in video captioning mainly focus on tailoring RNNs <cit.> or LSTMs <cit.> to generate better captions, assuming effective visual encoding of 𝒱 to be available in the form of `v'. The representation prowess of CNNs has made them the default choice for visual encoding in the existing literature. However, due to the nascency of video captioning research, only primitive methods of using CNN features for  `v' can be found in the literature.
These methods directly use  2D/3D CNN features or their concatenations for visual encoding, where the temporal dimension of the video is resolved by mean pooling <cit.>. 


We acknowledge the role of apt sequence modeling for video description, however, we also argue that designing specialized visual encoding techniques for captioning is equally important. Hence, 
we mainly focus on the operator 𝒬(.) in the mapping ℳ(𝒬( 𝒱))) →𝒮, where 𝒬( 𝒱) → v. We propose a visual encoding technique that, along harnessing the power of CNN features, explicitly encodes spatio-temporal dynamics of the scene in the visual representation, and embeds semantic attributes in it to further help the sequence modelling phase of video description to generate semantically rich textual sentences.



 §.§ Visual Encoding


For clarity, we describe the visual representation of a video 𝒱 as v = [α; β;  γ; η], where α to γ are themselves column-vectors computed by the proposed technique.
We explain these computations in the following. 





  §.§.§ Encoding Temporal Dynamics


In the context of video description, features extracted from pre-trained 2D-CNNs, e.g. VGG <cit.> and 3D-CNNs, e.g. C3D <cit.> have been shown useful for visual encoding of videos. The standard practice is to forward pass individual video frames through a 2D CNN and store  activation values of a pre-selected extraction layer of the network.
Then, perform mean pooling over those  activations for all the frames to compute the visual representation.
A similar procedure is adopted with 3D CNN with a difference that video clips are used in forward passes instead of   frames. 






A simple mean pooling operation over activation values is bound to fail in encoding  fine-grained temporal dynamics of the  video.
This is true for both 2D and 3D CNNs, despite the fact that the latter models video clips.
We address this shortcoming by defining  transformations 𝒯_f (ℱ) →α and 𝒯_c (𝒞) →β, such that ℱ = {_1, _2,..., _f} and 𝒞 = {_1, _2,..., _c}. Here, _t and _t denote the activation vectors of the extraction layers of 2D and 3D CNNs for the t^th video frame and video clip respectively. The aim of these transformations is to compute α and β that encode temporal dynamics of the complete video with high fidelity. 

We use the last avg pool layer of InceptionResnetV2 <cit.> to compute _i, and the fc6 layer of C3D <cit.> to get _i. The transformations 𝒯_f/c(.) are defined over the activations of those extraction layers. Below, we explain 𝒯_f(.) in detail. The transformation  𝒯_c(.) is similar, except that it uses activations of clips instead of   frames. 

Let a_j,t^i denote the activation value of the j^th neuron of the network's extraction layer for the t^th frame of the i^th training video. We leave out the superscript 2D for better readability. To perform the transform, we first define _1 a_j^i = [a_j,1^i, a_j,2^i,...,a_j,f^i] ∈ℝ^f and compute Ψ(_1 a_j^i) →ς_1 ∈ℝ^ p, where the operator Ψ(.) computes the Short Fourier Transform <cit.> of the vector in its argument and stores the first `p' coefficients of the transform. Then, we divide _1 a_j^i into two smaller vectors _21 a_j^i ∈ℝ^h and _22 a_j^i ∈ℝ^h-f, where h = ⌊f/2⌋. We again apply the operator  Ψ(.) to these vectors to compute ς_21 and ς_22 in p-dimensional space. We recursively perform  the same operations on ς_21 and ς_22 to get the p-dimensional vectors ς_311, ς_312 , ς_321, and ς_322. We combine all these vectors as ς (j) = [ς_1, ς_21, ς_22, ..., ς_322] ∈ℝ^(p × 7) × 1. We also illustrate this operation in Fig. <ref>. The same operation is performed individually for each neuron of our extraction layer. We then concatenate ς (j): j ∈{1,2,...,m} to form α∈ℝ^(p × 7 × m) × 1, where m denotes the number of neurons in the  extraction layer.  As a result of performing   𝒯_f(ℱ) →α, we have computed a representation the video while accounting for fine temporal dynamics in the whole sequence of video frames.     
Consequently,  𝒯_f(.) results in a much more informative representation than that obtained with mean pooling of the neuron activations.



We define 𝒯_c(.) in a similar manner for the set 𝒞 of video clip activations. This transformation results in β∈ℝ^(p × 7 × k) × 1, where k denotes the number of neurons in the extraction layer of the 3D CNN. It is worth mentioning that a 3D CNN is already trained on short video clips. Hence, its features  account for the temporal dimension of 𝒱 to some extent. Nevertheless, accounting for the fine temporal details in the whole video  adds to our  encoding significantly (see Section <ref>). It is noteworthy that exploiting  Fourier Transform in a hierarchical fashion to encode temporal dynamics has also been considered in human action recognition <cit.>. However, this work is the first to apply Short Fourier Transform hierarchically for video captioning.   





  §.§.§ Encoding Semantics and Spatial Evolution


It is well-established that the latter layers of CNNs are able to learn features at higher levels of abstraction due to hierarchical application of convolution operations in the earlier layers <cit.>. The common use of activations of e.g. fully-connected layers as visual features for captioning is also motivated by the fact that these representations are  discriminative transformations of high-level video features. We take this concept further and argue that the output layers of CNNs can themselves serve as discriminative encodings of the highest abstraction level for video captioning. 
We describe the  technique to effectively exploit these features  in the paragraphs to follow. Here, we briefly emphasize that the output layer of a network contains additional  information for video captioning beyond what is provided by the commonly used extraction layers of networks, because:


  * The output labels are yet another transformation of the extraction layer features, resulting from network weights that are  unaccounted for by  extraction layer.

  * The semantics attached to the output layer are at the same level of abstraction that is encountered in video  captions - a unique property of the output layers. 


We  use  the output layers of an Object Detector (i.e. YOLO <cit.>) and a 3D CNN (i.e. C3D <cit.>) to extract semantics pertaining to the objects and actions recorded in videos. The core idea is to quantitatively embed object labels, their frequencies of occurrence, and evolution  of their spatial locations in videos   in the visual encoding vector. Moreover, we also aim to enrich our visual encoding with the semantics of actions performed in  the video. The details of materializing  this concept are presented below. 




  
Objects Information:
Different from classifiers that only predict labels of input images/frames, object detectors can localize multiple objects in individual frames, thereby providing cues for ascertaining  plurality of the same type of objects in individual frames and  evolution of objects' locations in multiple frames. Effective embedding of such high-level information in vector `v' promises descriptions that can clearly differentiate between e.g. `people running' and `person walking' in a video. 

The sequence modeling component of a video captioning system generates a textual sentence  by selecting words from a large dictionary 𝒟. An object detector provides a set ℒ of object labels at its output. We first compute ℒ =  𝒟⋂ℒ, and define γ = [ζ_1, ζ_2, ..., ζ_|ℒ| ], where |.| denotes the cardinality of a set.
The vectors ζ_i, ∀ i in γ are further defined with the help `q' frames sampled from the original video. We perform this sampling using a fixed time interval between the sampled frames of  a given video. The samples are passed through the object detector and its output is utilized in computing ζ_i, ∀ i. 
A vector ζ_i is  defined as ζ_i = [Pr(ℓ_i), Fr(ℓ_i), ν_i^1,  ν_i^2,...,  ν_i^(q-1)], where ℓ_i indicates the i^th element of ℒ (i.e. an object name), Pr(.) and Fr(.) respectively compute the probability and frequency of occurrence of the object corresponding to ℓ_i, and ν_i^z represent the velocity of the object between the frames z and z+1 (in the sampled q frames).

We define γ  over `q' frames, whereas the used object detector processes individual frames that results in a probability and frequency value for each frame. We resolve this and related  mismatches by using the following definitions of the components of ζ_i: 


  * Pr(.) = max_z  Pr_z(.) : z ∈{1,...,q}.

  * Fr(.) = max_z  Fr_z(.)/N : z ∈{1,...,q}, where `N' is the allowed  maximum number of the same class of objects detected in a frame. We let N =10 in experiments.

  * ν_i^z = [δ_x^z, δ_y^z] : δ_x^z = x̃^z+1 - x̃^z and δ_y^z = ỹ^z+1 - ỹ^z. Here, x̃, ỹ denote the Expected values of the x and y coordinates of the same type of objects in a given frame, such that the coordinates are also normalized by the respective frame dimensions. 


We let q = 5 in our experiments, resulting in ζ_i ∈ℝ^10, ∀ i that compose γ∈ℝ^(10 × |ℒ|) × 1.  The indices of coefficients in γ identify the object labels in videos (i.e. probable nouns to appear in the description). Unless an object is detected in the video, the coefficients of γ corresponding to it are kept zero. 
The proposed embedding of  high level semantics in γ contain highly relevant information about objects in explicit form for a sequence learning module of video description system. 





  
Actions Information: Videos generally record  objects and their interaction. The latter is best described by the actions performed in the videos. We already use a 3D CNN that learns action descriptors for the videos. We tap into the output layer of that network to further embed high level action information in our visual encoding. To that end, we compute 𝒜 = 𝒜⋂𝒟, where 𝒜 is the set of labels at the output of the 3D CNN. Then, we define η = [ [ϑ_1, Pr(ℓ_1)],   [ϑ_2, Pr(ℓ_2)], ...,  [ϑ_|𝒜|, Pr(ℓ_|𝒜|)]] ∈ℝ^(2 × |𝒜|) × 1, where ℓ_i is the i^th element of 𝒜 (an action label) and ϑ is a binary variable that is 1 only if the action is predicted by the network. 

We concatenate the above described vectors α, β, γ and η to form our visual encoding vector v∈ℝ^d, where d = 2× (p × 7 × m) + (10 × |ℒ|) + (2 × |𝒜|). Before passing this vector to a sequence modelling component of our method, we perform its compression using a fully connected layer, as shown in Fig. <ref>. Using tanh activation function and fixed weights, this  layer projects `v' to a 2K-dimensional space. The resulting projection `υ'  is used by our language model.  



 §.§ Sequence Modelling



We follow the common pipeline of video description techniques that feeds visual representation of a video to a sequence modelling component, see Fig. <ref>. Instead of resorting to a sophisticated language model, we develop a relatively simpler model employing multiple layers of  Gated Recurrent Units (GRUs) <cit.>.  
GRUs are known to be more robust to vanishing gradient problem - an issue encountered in long captions - due to their ability of remembering the relevant information and forgetting the rest over time.

A GRU has two gates: reset Γ_r and update Γ_u, where the  update gate decides how much the unit updates its previous memory and the reset gate determines how to combine the new input with the previous memory.
Concretely, our language model computes the hidden state h^<t> of a GRU as:




    Γ_u = σ(W_u[h^<t-1>, x^<t>] + b_u)




    Γ_r = σ(W_r[h^<t-1>, x^<t>] + b_r)




    h̃^<t> = tanh(W_h[Γ_r ⊙ h^<t-1>, x^<t>] + b_h




    h^<t> = Γ_u ⊙h̃^<t> + (1 - Γ_u) ⊙ h^<t-1>
 


where, ⊙ denotes the hadamard product, σ(.) is sigmoid activation , W_q, ∀ q are learnable weight matrices, and b_u/r/h denote the respective biases. In our approach, h^<0> = υ for a given video, whereas the signal x is the word embedding vector. 
In Section <ref>, we report results using two layers of GRUs, and demonstrate that our language model under the proposed straightforward sequence modelling  already provides highly competitive performance due to the proposed visual encoding. 




§ EXPERIMENTAL EVALUATION







 §.§ Datasets

We evaluate our technique using two popular benchmark datasets from the existing literature in video description, namely Microsoft Video Description (MSVD) dataset <cit.>, and MSR-Video To Text (MSR-VTT) dataset <cit.>.  We first give details of these datasets and their processing performed in this work, before discussing the experimental results.  



MSVD Dataset <cit.>: This dataset is composed of  1,970 YouTube open domain videos that predominantly show only a single activity each. Generally, each clip is spanning  over 10 to 25 seconds. The dataset provides multilingual human annotated sentences as captions for the videos. We experiment with the captions in English. On average, 41 ground truth captions can be associated with a single video. For benchmarking, we follow the common data split of 1,200 training samples, 100 samples for validation and 670 videos for testing <cit.>. 



MSR-VTT Dataset <cit.>:
This recently introduced open domain videos dataset contains a wide variety of videos for the captioning task. It consists of 7,180 videos that are transformed into 10,000 clips. The clips are grouped into 20 different categories. Following the common settings <cit.>, we divide the 10,000 clips into 6,513 samples for training, 497 samples for validation and the remaining 2,990 clips for testing. Each video is described by 20 single sentence annotations by Amazon Mechanical Turk  (AMT) workers. This is one of the largest clips-sentence pair dataset available for the video captioning task, which is the main  reason of choosing this dataset for benchmarking our technique. 






 §.§ Dataset Processing & Evaluation Metrics
 


We converted the captions in both datasets to lower case, and removed all punctuations. All the sentences were then tokenized. We set the vocabulary size for MSVD to  9,450 and for MSR-VTT to 23,500. We employed  “fasttext“ <cit.> word embedding vectors of dimension 300. Embedding vectors of 1,615 words for MSVD and 2,524 words for MSR-VTT were not present in the pretrained set. Instead of using randomly initialized vectors or ignoring the out of vocabulary words entirely in the training set, we generated embedding vectors for these words using character n-grams within the word, and summing the resulting vectors to produce the final vector. We performed dataset specific fine-tuning on the pretrained word embeddings.


In order to compare our technique with the existing methods, we report results on the four most popular metrics, including; Bilingual Evaluation Understudy (BLEU) <cit.>, Metric for Evaluation of Translation with Explicit Ordering
(METEOR) <cit.>, Consensus based Image Description Evaluation (CIDEr_D) <cit.> and Recall Oriented
Understudy of Gisting Evaluation (ROUGE_L) <cit.>. We refer to the original works for the concrete definitions of these metrics. The subscript `D' in CIDEr indicates the metric variant that inhibits higher values for inappropriate captions in human judgment. Similarly, the subscript `L' indicates the variant of ROUGE  that is based on recall-precision scores of the longest common sequence between  the prediction and the ground truth.

We used the Microsoft COCO server <cit.> to compute our results. 





 §.§ Experiments


In our experiments reported below[Due to through evaluation, supplementary material also contains further results. Only the best performing setting  is discussed here.], we use  InceptionResnetV2 (IRV2) <cit.> as the 2D CNN, whereas C3D <cit.> is  used as the 3D CNN.
The last `avg pool' layer of the former, and the `fc6' layer of the latter are considered as the extraction layers.
The 2D CNN is pre-trained on the popular ImageNet dataset <cit.>, whereas Sports 1M dataset <cit.> is used for the pre-training of C3D. To process videos, we re-size the frames to match the input dimensions of these networks.
For the 3D CNN, we use 16-frame clips as inputs with an 8-frame overlap. 
YOLO <cit.> is used as the object detector in all our experiments. 
To train our language model, we include a start and an end token to the captions to deal with the dynamic length of different sentences.
We set the maximum sentence length to 30 words in the case of experiments with MSVD dataset, and to 50 for the MSR-VTT dataset. 
These length limits are based on the available captions in the datasets. We truncate a sentence if its length exceeds the set limit, and zero pad in the case of shorter length. 



 We tune the hyper-parameters of our language model on the validation set. The results below use two layers of GRUs, that employ 0.5 as the dropout value. We use the RMSProp algorithm with a learning rate 2 × 10^-4 to train the models. A batch size of 60 is used for training in our experiments. We performed training of our models for 50 epochs. We used the sparse cross entropy loss to train our model.
 The training is conducted using NVIDIA Titan XP 1080 GPU. We used TensorFlow framework for development. 
 





  §.§.§ Results on MSVD dataset






We comprehensively benchmark our method against the current state-of-the-art in video captioning. We report the results of the existing methods and our approach in Table. <ref>.  For the existing techniques, recent best performing methods are chosen and their results are directly taken from the existing literature (same evaluation protocol is ensured).
The table columns present scores for the metrics BLEU-4 (B-4), METEOR (M), CIDEr_D (C) and ROUGE_L (R). 


The last seven rows of the Table report results of different variants of our  method to highlight the contribution of various components of the overall technique. GRU-MP indicates that we use our two-layer GRU model, while the common `Mean Pooling (MP)' strategy is adopted to resolve the temporal dimension of videos. `C3D' and `IRV2' in the parentheses identify  the networks used to compute the visual codes. We abbreviate the joint use of C3D and IRV2 as `CI'. We use `EVE' to denote our Enriched Visual Encoding that applies Hierarchical Fourier Transform - indicated by  the subscript `hft' - on the activations of the network extraction layers. The proposed final technique, that also incorporates the high-level semantic information - indicated by the subscript `+sem' - is mentioned in the last row of the Table. We also follow the same notational conventions for our method in the remaining Tables.   







Our method achieves a strong 35 value of METEOR, which provides a 35.0 - 34.1/34.1× 100 = 2.64% gain over the closest competitor. Similarly,  gain over the current state-of-the-art for ROUGE_L is 2.44%. For the other metrics, our scores  remain competitive to the best performing methods. It is emphasized, that our approach derives its main strength from the visual encoding part in contrast to sophisticated language model, which is generally the case for the existing methods. Naturally, complex language models entail difficult and computationally expensive training process, which is not a limitation of our approach.   

We illustrate representative qualitative results of our method in Fig. <ref>. We abbreviate our final approach as `GRU-EVE' in the figure for brevity. The semantic details and accuracy of  e.g. plurality, nouns and verbs is clearly visible in the captions generated by the proposed method. The figure also reports the captions for GRU-MP-(CI) and GRU-EVE_hft-(CI) to show the difference resulting from hierarchical Fourier transform (hft) as compared to the Mean Pooling (MP) strategy. These captions justify the noticeable gain achieved by the proposed hft over the traditional MP in Table <ref>. We also observe in the table that our method categorically outperforms the mean pool based methods, i.e. LSTM-YT <cit.>, LSTM-E <cit.>, SCN-LSTM <cit.>, and LSTM-TSA<cit.> on METEOR, CIDEr and ROUGE_L. Under these observations, we safely recommend the proposed hierarchical Fourier transformation as the substitute for the `mean pooling' in video captioning.    



In Table <ref>, we compare the variant of our method based on a single CNN with the best performing single CNN based existing methods. The results are directly taken from <cit.> for the provided METEOR metric. As can be seen, our method outperforms all these methods.  In Table <ref>, we also compare our method on METEOR with the state-of-the-art methods that necessarily use  multiple visual features to obtain the best performance. A significant 5.1% gain is achieved by our method to the closest competitor in this regard.   


























  §.§.§ Results on MSR-VTT dataset


MSR-VTT <cit.> is a recently released dataset. We compare performance of our approach on this dataset with the latest published models such as Alto <cit.>, RUC-UVA <cit.>, TDDF <cit.>, PickNet <cit.>, M^3-VC <cit.> and RecNet_local <cit.>. The results are summarized in Table <ref>. 
Similar to the MSVD dataset, our method significantly improves the state-of-the-art on this dataset on METEOR and ROUGE_L metrics, while achieving strong results on the remaining metrics. 
These result ascertain the effectiveness of the proposed enriched visual encoding for visual captioning. We provide examples of qualitative results on this dataset in the supplementary material of the paper.












§ DISCUSSION




We conducted a through empirical evaluation of the proposed method to explore its different aspects. Below we discuss and highlight few of these aspects in the text. Where necessary, we also provide results in the supplementary material of the paper to back the discussion.    

For the settings discussed in the previous section, we generally observed semantically rich captions generated by the proposed approach. In particular, these captions well captured the plurality of objects and their motions/actions. Moreover, the captions generally described the whole videos instead of its partial clips. Instead of only two, we also tested different number of GRU layers, and observed that  increasing the number of GRU layers deteriorated the BLEU-4 score. However, there were improvements in all the remaining metrics. We retained only two GRU layers in the final method mainly for computational gains. Moreover, we also tested different architectures of GRU, e.g. with state sizes 512, 1024, 2048 and 4096. We observed a trend of performance improvement until 2048 states. However, further states did not improve the performance. Hence, 2048 were finally used in the results reported in the previous section. 

Whereas all the components of the proposed technique contributed to  the overall final performance, the biggest revelation of our work is the use of hierarchical Fourier Transform to capture the temporal dynamics of videos. As compared to the `nearly standard' mean pooling operation performed in the existing captioning pipeline, the proposed use of Fourier Transform promises a  significant performance gain for any method. Hence, we safely recommend replacing the mean pooling operation with our transformation for the future techniques.    


















§ CONCLUSION



We presented a novel technique for visual encoding of videos to generate semantically rich captions. Besides capitalizing on the representation power of CNNs, our method explicitly accounts for the spatio-temporal dynamics of the scene, and high-level semantic concepts encountered in the video. We applying Short Fourier Transform to 2D and 3D CNN features of the videos in a hierarchical manner, and account for the high-level semantics by processing output layer features of an Object Detector and the 3D CNN. Our enriched visual representation is used to learn a relatively simple GRU based language model that performs on-par or better than   the existing video description methods on popular MSVD and MSR-VTT datasets. 

Acknowledgment This research was supported by ARC Discovery Grant DP160101458 and partially by DP190102443. The Titan XP GPU used in our experiments was donated by NVIDIA corporation.

ieee



