Memory-augmented Attention Modelling for Videos
Rasool Fakoor†∗, Abdel-rahman Mohamed,†† , Margaret Mitchell‡† , Sing Bing Kang†† ,
Pushmeet Kohli††
††
†
Microsoft Research University of Texas at Arlington ‡† Google

arXiv:1611.02261v4 [cs.CV] 24 Apr 2017

†

rasool.fakoor@mavs.uta.edu, †† {asamir, singbing.kang, pkohli}@microsoft.com ‡† mmitchellai@google.com,

Abstract

that refer to actions and events taking place over multiple
frames in a video. For example, in a video depicting “someone waving a hand”, the “waving” action can start from any
frame and can continue on for a variable number of following frames. At the same time, videos contain many frames
that do not provide additional information over the smaller
set of frames necessary to generate a summarizing description. Given these challenges, it is not surprising that even
with recent advancements in image captioning (Fang et al.,
2015; Xu et al., 2015a; Johnson et al., 2016; Vinyals et al.,
2015; Donahue et al., 2015), video captioning has remained
challenging.

We present a method to improve video description generation by modeling higher-order interactions between video frames and described concepts. By storing past visual attention in the
video associated to previously generated words,
the system is able to decide what to look at and
describe in light of what it has already looked
at and described. This enables not only more
effective local attention, but tractable consideration of the video sequence while generating each
word. Evaluation on the challenging and popular MSVD and Charades datasets demonstrates
that the proposed architecture outperforms previous video description approaches without requiring external temporal video features. The source
code for this paper is available on https://
github.com/rasoolfa/videocap.

Motivated by these observations, we introduce a memorybased attention mechanism for video captioning and description. Our model utilizes memories of past attention
in the video when reasoning about where to attend in a
current time step. This allows the model to not only effectively leverage local attention, but also to consider the
entire video as it generates each word. This mechanism effectively binds information from both vision and language
sources into a coherent structure.

1. Introduction
Deep neural architectures have led to remarkable progress
in computer vision and natural language processing problems. Image captioning is one such problem, where the
combination of convolutional structures (Krizhevsky et al.,
2012; LeCun et al., 1998), and sequential recurrent structures (Sutskever et al., 2014) leads to remarkable improvements over previous work (Fang et al., 2015; Devlin et al.,
2015). One of the emerging modelling paradigms, shared
by models for image captioning as well as related visionlanguage problems, is the notion of an attention mechanism
that guides the model to attend to certain parts of the image
while generating (Xu et al., 2015a).

Our work shares the same goals as recent work on attention mechanisms for sequence-to-sequence architectures,
such as Rocktäschel et al. (2016) and Yang et al. (2016).
Rocktäschel et al. (2016) consider the domain of entailment
relations, where the goal is to determine entailment given
two input sentences. They propose a soft attention model
that is not only focused on the current state, but the previous
as well. In our model, all previous attentions are explicitly
stored into memory, and the system learns to memorize the
encoded version of the input videos conditioned on previously seen words. Yang et al. (2016) and our work both
try to solve the problem of locality of attention in visionto-language, but while Yang et al. (2016) introduce a memory architecture optimized for single image caption generation, we introduce a memory architecture that operates on
a streaming video’s temporal sequence.

The attention models used for problems such as image captioning typically depend on the single image under consideration and the partial output generated so far, jointly
capturing one region of an image and the words being generated. However, such models cannot directly capture the
temporal reasoning necessary to effectively produce words
∗
Corresponding author:
sool.fakoor@mavs.uta.edu)

Rasool

Fakoor

(

The contributions of this work include:

ra-

1

• A deep learning architecture that represents video
with an explicit model of the video’s temporal struc-

ture.
• A method to jointly model the video description and
temporal video sequence, connecting the visual video
space and the language description space.
• A memory-based attention mechanism that learns iterative attention relationships in a simple and effective
sequence-to-sequence memory structure.
• Extensive comparison of this work and previous work
on the video captioning problem on the MSVD (Chen
& Dolan, 2011) and the Charades (Sigurdsson et al.,
2016) datasets.

which frames are most salient. Further, the models need
additional work to be able to focus on points of interest
within the video frames to select what to talk about. Even
a variable-length vector to represent a video using attention
(Yao et al., 2015) can have some problems.
More specifically, current attention methods are local (Yang et al., 2016), since the attention mechanism
works in a sequential structure, and lack the ability to capture global structure. Moreover, combining a video and a
description as a sequence-to-sequence problem motivates
using some variant of a recurrent neural network (RNN)
(Hochreiter & Schmidhuber, 1997): Given the limited capacity of a recurrent network to model very long sequences,
memory networks (Weston et al., 2014; Sukhbaatar et al.,
2015) have been introduced to help the RNN memorize sequences. However, one problem these memory networks
suffer from is difficulty in training the model. The model
proposed by Weston et al. (2014) requires supervision at
each layer, which makes training with backpropagation a
challenging task. Sukhbaatar et al. (2015) proposed a memory network that can be trained end-to-end, and the current
work follows this research line to tackle the challenging
problem of modeling vision and language memories for
video description generation.

We focus on the video captioning problem, however, the
proposed model is general enough to be applicable in other
sequence problems where attention models are used (e.g.,
machine translation or recognizing entailment relations).

2. Related Work
One of the primary challenges in learning a mapping from
a visual space (i.e., video or image) to a language space
is learning a representation that not only effectively represents each of these modalities, but is also able to translate a representation from one space to the other. Rohrbach
et al. (2013) developed a model that generates a semantic representation of visual content that can be used as
the source language for the language generation module.
Venugopalan et al. (2015b) proposed a deep method to
translate a video into a sentence where an entire video is
represented with a single vector based on the mean pool
of frame features. However, it was recognized that representing a video by an average of its frames loses the temporal structure of the video. To address this problem, recent work (Yao et al., 2015; Pan et al., 2016a; Venugopalan
et al., 2015a; Shin et al., 2016; Pan et al., 2016b; Xu et al.,
2015b; Ballas et al., 2016; Yu et al., 2016) proposed methods to model the temporal structure of videos as well as
language.
The majority of these methods are inspired by sequence-tosequence (Sutskever et al., 2014) and attention (Bahdanau
et al., 2015) models. Sequence learning was proposed to
map the input sequence of a source language to a target
language (Sutskever et al., 2014). Applying this method
with an additional attention mechanism to the problem of
translating a video to a description showed promising initial results, however, revealed additional challenges. First,
modelling the video content with a fixed-length vector in
order to map it to a language space is a more complex problem than mapping from a language to a language, given the
complexity of visual content and the difference between the
two modalities. Since not all frames in a video are equally
salient for a short description, and an event can happen
in multiple frames, it is important for a model to identify

3. Learning to Attend and Memorize
A main challenge in video description is to find a mapping
that can capture the connection between the video frames
and the video description. Sequence-to-sequence models,
which work well at connecting input and output sequences
in machine translation (Sutskever et al., 2014), do not perform as well for this task, as there is not the same direct
alignment between a full video sequence and its summarizing description.
Our goal in the video description problem is to create an
architecture that learns which moments to focus on in a
video sequence in order to generate a summarizing natural language description. The modelling challenges we set
forth for the video description problem are: (1) Processing
the temporal structure of the video; (2) Learning to attend
to important parts of the video; and (3) Generating a description where each word is relevant to the video. At a
high-level, this can be understood as having three primary
parts: When moments in the video are particularly salient;
what concepts to focus on; and how to talk about them.
We directly address these issues in an end-to-end network
with three primary corresponding components (Figure 1):
A Temporal Model (TEM), An Iterative Attention/Memory
Model (IAM), and a Decoder. In summary:

2

• When: Frames within the video sequence - The Temporal Model (TEM).

(b)

IAM

(a)

Decoder
h1v

h2v

h3v

hNv

h0g

A

ht’-1g
walking

ht’g

hTg

into

glass

TEM

#

h1g

Figure 1. Our proposed architecture. Each component of our model is described in 3.1 through 3.3.

• What: Language-grounded concepts depicted in the
video - The Iterative Attention/Memory mechanism
(IAM).

is a Recurrent Neural Network (RNN), which has been
shown to be effectual in modelling the temporal structure of sequential data such as video (Ballas et al., 2016;
Sharma et al., 2015; Venugopalan et al., 2015a) and speech
(Graves & Jaitly, 2014). In order to apply this in video
sequences to generate a description, we seek to capture
the fact that frame-to-frame temporal variation tends to be
local (Brox & Malik, 2011) and critical in modeling motion (Ballas et al., 2016). Visual features extracted from
the last fully connected layers of Convolutional Neural Networks (CNNs) have been shown to produce state-of-the-art
results in image classification and recognition (Simonyan
& Zisserman, 2014; He et al., 2016), and thus seem a good
choice for modeling visual frames. However, these features
tend to discard low level information useful in modeling the
motion in the video (Ballas et al., 2016).

• How: Words that fluently describe the what and when
- The Decoder.
The Temporal Model is in place to capture the temporal
structure of the video: It functions as a when component.
The Iterative Attention/Memory is a main contribution of
this work, functioning as a what component to remember
relationships between words and video frames, and storing
longer term memories. The Decoder generates language,
and functions as the how component to create the final description.
To train the system end to end, we formulate the problem
as sequence learning to maximize the probability of generating a correct description given a video:
X
Θ∗ = arg max
log p(S|f1 , . . . , fN ; Θ) (1)
Θ

To address these challenges, we implement an RNN we call
the Temporal Model (TEM). At each time step of the TEM, a
video frame encoding from a CNN serves as input. Rather
than extracting video frame features from a fully connected
layer of the pretrained CNN, we extract intermediate convolutional maps.

(S,f1 ,...,fN )

where S is the description, f1 , f2 , . . . , fN are the input
video frames, and Θ is the model parameter vector. In
the next sections, we will describe each component of the
model, then explain the details of training and inference.

In detail, for a given video X with N frames X =
[X 1 , X 2 , · · · , X N ], N convolutional maps of size RL×D
are extracted, where L is the number of locations in the input frame and D is the number of dimensions (See TEM
in Figure 1). To enable the network to store the most important L locations of each frame, we use a soft location attention mechanism, fLatt (Bahdanau et al., 2015; Xu et al.,
2015a; Sharma et al., 2015). We first use a softmax to compute L probabilities that specify the importance of different
parts in the frame, and this creates an input map for fLatt .

Notational note: Numbered equations use bold face to denote
multi-dimensional learnable parameters, e.g., Wpj . To distinguish
the two different sets of time steps, one for video frames and one
for words in the description, we use the notation t for video and t0
for language. Throughout, the terms description and caption are
used interchangeably.

Formally, given a video frame at time t, X t ∈ RL×D , the

3.1. Temporal Model (TEM)
The first module we introduce encodes the temporal structure of the input video. A clear framework to use for this

3

fLatt mechanism is defined as follows:

• Given:

exp(Wpj ht−1
v )
ρtj = PL
k t−1
k=1 exp(Wp hv )
fLatt (X

t

, ht−1
v ; Wp )

=

N = Number of frames in a given video

(2)

L
X

T = Number of words in description
Hv = Input video states, [h1v , ..., hN
v ]

ρtj Xjt

0

Hgt −1 = Decoder state hg at time t-1, repeated N times

(3)

j=1

0

t −1
Hm
= Memory state hm at time t-1, repeated N times

where ht−1
∈ RK is the hidden state of the TEM at time
v
t-1 with K dimensions, and Wp ∈ RL×K . For each video
frame time step, TEM learns a vector representation by applying location attention on the frame convolution map,
conditioned on all previously seen frames:
F t = fLatt (X t , ht−1
v ; Wp )

(4)

htv

(5)

t

= fv (F ,

ht−1
v ; Θv )

Wv , Wg ∈ RK×K
Wm ∈ RM ×K
u ∈ RK
α = Probability over all N frames
Θa = {Wv , Wg , Wm , u}

• Attention update [F̂ (Θa )]:
0

where fv can be an RNN/LSTM/GRU cell and Θv is the
parameters of the fv . Due to the fact that vanilla RNNs
have gradient vanishing and exploding problems (Pascanu
et al., 2013), we use gradient clipping, and an LSTM with
the following flow to handle potential vanishing gradients:

0

t −1
QA = tanh(Hv Wv + Hgt −1 Wg + Hm
Wm )
αt0 = softmax(QA u)

(6)
(7)

F̂ = HvT αt0

(8)

• Memory update:
0

0

htm = fm (htm−1 , F̂ ; Θm )

T
it = σ(F t Wxi + (ht−1
v ) Whi )
T
f t = σ(F t Wxf + (ht−1
v ) Whf )

(9)

Figure 2. Iterative Attention and Memory (IAM) is formulated as
an Attention update and a Memory update.

T

ot = σ(F t Wxo + (ht−1
v ) Who )
T

g t = tanh(F t Wxg + (ht−1
v ) Whg )
ctv = f t

ct−1
+ it
v

htv = ot

tanh(ct )

This component addresses several key issues in generating
a coherent video description. In video description, a single word or phrase often describes action spanning multiple frames within the input video. By employing the IAM,
the model can effectively capture the relationship between
a relatively short bit of language and an action that occurs
over multiple frames. This also functions to directly address the problem of identifying which parts of the video
are most relevant for description.

gt

where Wh∗ ∈ RK×K , Wx∗ ∈ RD×K , and we define Θv =
{Wh∗ , Wx∗ }.
3.2. Iterative Attention/Memory (IAM)
A main contribution of this work is a global view for the
video description task: A memory-based attention mechanism that learns iterative attention relationships in an efficient sequence-to-sequence memory structure. We refer to
this as the Iterative Attention/Memory mechanism (IAM),
and it aggregates information from previously generated
words and all input frames.
The IAM component is an iterative memorized attention between an input video and a description. More specifically,
it learns a iterative attention structure for where to attend in
a video given all previously generated words (from the Decoder), and previous states (from the TEM). This functions
as a memory structure, remembering encoded versions of
the video with corresponding language, and in turn, enabling the Decoder to access the full encoded video and
previously generated words as it generates new words.

The proposed Iterative Attention/Memory mechanism is
formalized with an Attention update and a Memory update, detailed in Figure 2. Figure 1 illustrates where the
IAM sits within the full model, with the Attention module
shown in 1a and the Memory module shown in 1b.

4

As formalized in Figure 2, the Attention update F̂ (Θa )
computes the set of probabilities in a given time step for
attention within the input video states, the memory state,
and the decoder state. The Memory update stores what has
been attended to and described. This serves as the memorization component, combining the previous memory with
the current iterative attention F̂ . We use an LSTM fm
with the equations described above to enable the network to
learn multi-layer attention over the input video and its corresponding language. The output of this function is then

used as input to the Decoder.

MSVD is a set of YouTube videos annotated by workers
on Mechanical Turk,2 who were asked to pick a video clips
representing an activity. In this dataset, each clip is annotated by multiple workers with a single sentence. The
dataset contains 1, 970 videos and about 80, 000 descriptions, where 1, 200 of the videos are training data, 670 test,
and the rest (100 videos) for validation. In order for the
results to be comparable to other approaches, we follow
the exact training/validation/test splits provided by Venugopalan et al. (2015b).

3.3. Decoder
In order to generate a new word conditioned on all previous
words and IAM states, a recurrent structure is modelled as
follows:
0

0

0

0

htg = fg (st , htm , hgt −1 ; Θg )
0

0

ŝt = softmax((htg )T We )
0

(10)
(11)

0

where htg ∈ RK , st is a word vector at time t0 , We ∈
RK×|V | , and |V | is the vocabulary size. In addition, ŝt0
assigns a probability to each word in the language. fg is an
0
0
0
LSTM where st and htm are inputs and htg is the recurrent
state.

Evaluation metrics We report results on the video description generation task. In order to evaluate descriptions
generated by our model, we use model-free automatic evaluation metrics. We adopt METEOR, BLEU - N, and CIDEr
metrics available from the Microsoft COCO Caption Evaluation code3 to score the system.

3.4. Training and Optimization
Video and Caption preprocessing We preprocess the
captions for both datasets using the Natural Language
Toolkit (NLTK)4 and clip each description up to 30 words,
since the majority have less. We extract sample frames
from each video and pass each frame through VGGnet (Simonyan & Zisserman, 2014) without fine-tuning. For the
experiments in this paper, we use the feature maps from
conv5 3 layer after applying ReLU. The feature map in this
layer is 14×14×512. Our TEM component operates on the
flattened 196 × 512 of this feature cubes. For the ablation
studies, features from the fully connected layer with 4096
dimensions are used as well.

The goal in our network is to predict the next word
given all previously seen words and an input video.
In order to optimize our network parameters Θ =
{Wp , Θv , Θa , Θm , Θg , We }, we minimize a negative log
likelihood loss function:
L(S, X; Θ) = −

|V |
T X
X
t0

0

0

sti log(ŝti ) + λ k Θ k22

(12)

i

where |V | is the vocabulary size. We fully train our network in an end-to-end fashion using first-order stochastic gradient-based optimization method with an adaptive
learning rate. More specifically, in order to optimize our
network parameters, we use Adam (Kingma & Ba, 2015)
with learning rate 2 × 10−5 and set β1 , β2 to 0.8 and
0.999, respectively. During training, we use a batch size
of 16. The source code for this paper is available on
https://github.com/rasoolfa/videocap.

Hyper-parameter optimization We use random
search (Bergstra & Bengio, 2012) on the validation set
to select hyper-parameters on both datasets. The wordembedding size, hidden layer size (for both the TEM and
the Decoder), and memory size of the best model on
Charades are: 237, 1316, and 437, respectively. These
values are 402, 1479, and 797 for the model on the MSVD
dataset. A stack of two LSTMs are used in the Decoder and
TEM . The number of frame samples is a hyperparameter
which is selected among 4, 8, 16, 40 on the validation set.
ATT + N O TEM and N O IAM + TEM get the best results on
the validation set with 40 frames, and we use this as the
number of frames for all models in the ablation study.

4. Experiments
Dataset We evaluate the model on the Charades (Sigurdsson et al., 2016) dataset and the Microsoft Video Description Corpus (MSVD) (Chen & Dolan, 2011). Charades contains 9, 848 videos (in total) and provides 27, 8471
video descriptions. We follow the same train/test splits as
Sigurdsson et al. (2016), with 7569 train, 1, 863 test, and
400 validation. A main difference between this dataset and
others is that it uses a “Hollywood in Homes” approach to
data collection, where “actors” are crowdsourced to act out
different actions. This yields a diverse set of videos, with
each containing a specific action.

4.1. Video Caption Generation
We first present an ablation analysis to elucidate the contribution of the different components of our proposed model.
Then, we compare the overall performance of our model to
other recent models.
2

1

Only 16087 out of 27, 847 are used as descriptions for our
evaluation since the 27, 847 refers to script of the video as well as
descriptions.

https://www.mturk.com/mturk/welcome
https://github.com/tylin/coco-caption
4
http://www.nltk.org/
3

5

Ablation Analysis

The improvement over previous work is particularly noteworthy because we do not use external features for the
video, such as Optical Flow (Brox et al., 2004) (denoted Flow), 3-Dimensional Convolutional Network features (Tran et al., 2015) (denoted C3D), or fine-tuned CNN
features (denoted FT), which further enhances aspects such
as action recognition by leveraging an external dataset such
as UCF-101. The only system using external features that
outperforms the model proposed here is from Yu et al.
(2016), who uses a slightly different version of the same
dataset6 along with C3D features for a large improvement
in results (compare Table 2 rows 4 and 11); future work
may explore the utility of external visual features for this
work. Here, we demonstrate that the proposed architecture maps visual space to language space with improved
performance over previous work, before addition of further
resources.

Ablation results are shown in Table 1, evaluating on the
MSVD test set. The first (ATT + N O TEM) corresponds
to a simpler version of our model in which we remove the
TEM component and instead pass each frame of the video
through a CNN, extracting features from the last fullyconnected hidden layer. In addition, we replace our IAM
with a simpler version where the model only memorizes
the current step instead of all previous steps. In the next
variation (ATT + TEM), it is same as the first one except
we use TEM instead of fully connected CNN features. In
the next ablation (N O IAM + TEM), we remove the IAM
component from our model and keep the rest of the model
as-is. In the next variation (IAM + N O TEM), we remove
the TEM and calculate features for each frame, similar to
ATT + N O TEM. Finally, the last row in the table is our
proposed model (IAM + TEM) with all its components.

We additionally report results on the Charades dataset (Sigurdsson et al., 2016), which is challenging to train on because there are only a few (≈ 2) captions per video. In this
experiment, we use 16 frames per video as the input to the
TEM module. As shown in Table 3, our method achieves
a 10% relative improvement over the Venugopalan et al.
(2015a) model reported by Sigurdsson et al. (2016). It is
worth noting that humans reach a METEOR score of 24 and
a BLEU -4 score of 20, illustrating the low upper bound in
this task.7

The IAM plays a significant role in the proposed model, and
removing it causes a large drop in performance, as measured by both BLEU and METEOR. On the other hand, removing the TEM by itself does not drop performance as
much as dropping the IAM. Putting the two together, they
complement one another to result in overall better performance for METEOR. However, further development on the
TEM component in future work is warranted. In the N O
IAM + TEM condition, an entire video must be represented
with a fixed-length vector, which may contribute to the
lower performance (Bahdanau et al., 2015). This is in contrast to the other models, which apply single layer attention
or IAM to search relevant parts of the video aligned with
the description.

Results Discussion
We show some example descriptions generated by our system in Figure 3. The model generates mostly correct descriptions, with naturalistic variation from the ground truth.
Errors illustrate a preference to describe items that have a
higher likelihood of being mentioned, even if they appear
in less of the frames. For example, in the “a dog is on
a trampoline” video, our model focuses on the man, who
appears in only a few frames, and generates the incorrect
description “a man is washing a bath”. The errors, alongside the ablation study shown in Table 1, suggest that the
TEM module in particular may be further improved by focusing on how frames in the video sequence are captured
and passed to the IAM module.

Performance Comparison
To extensively evaluate the proposed model, we compare
with state-of-the-art models and baselines for the video
caption generation task on the MSVD dataset. In this experiment, we use 8 frames per video as the inputs to the
TEM module. As shown in Table 2,5 our proposed model
achieves state-of-the-art scores in BLEU-4, and outperforms
almost all systems on METEOR. The closest-scoring comparison system, from Pan et al. (2016a), shows a trade-off
between METEOR and BLEU: BLEU prefers descriptions
with short-distance fluency and high lexical overlap with
the observed descriptions, while METEOR permits less direct overlap and longer descriptions. A detailed study of
the generated descriptions between the two systems would
be needed to better understand these differences.

5. Conclusion
We introduce a general framework for an memory-based
sequence learning model, trained end-to-end. We apply this
framework to the task of describing an input video with a

5
The symbol − indicates that the score was not reported by
the corresponding paper. The horizontal line in Table 2 separates
models that do/do not use external features for the video representation.

6
Yu et al. (2016) uses the MSVD dataset reported in (Guadarrama et al., 2013), which has different preprocessing.
7
For comparison, the upper bound BLEU score in machine
translation for English to French is above 30.

6

Method

METEOR

BLEU -1

BLEU -2

BLEU -3

BLEU -4

CIDE r

ATT + N O TEM
ATT + TEM
N O IAM + TEM
IAM + N O TEM
IAM + TEM [40 F ]
IAM + TEM [8 F ]

31.20
31.00
30.50
31.00
31.70
31.80

77.90
79.00
78.10
78.70
79.00
79.40

65.10
66.50
65.20
66.90
66.20
67.10

55.30
56.30
55.10
57.40
56.0
56.80

44.90
45.50
44.60
47.00
45.60
46.10

63.90
61.00
60.50
62.10
62.20
62.70

Table 1. Ablation of proposed model with and without the IAM component on the MSVD test set.
Method

METEOR

BLEU -1

BLEU -2

BLEU -3

BLEU -4

CIDEr

Venugopalan et al. (2015b)
Venugopalan et al. (2015a)
Pan et al. (2016b)
Yu et al. (2016)
Pan et al. (2016a)
Our Model

27.7
29.2
29.5
31.10
33.10
31.80

−
−
74.9
77.30
79.20
79.40

−
−
60.9
64.50
66.30
67.10

−
−
50.6
54.60
55.10
56.80

−
−
40.2
44.30
43.80
46.10

−
−
−
−
−
62.70

Yao et al. (2015) + C3D
Venugopalan et al. (2015a) + Flow
Ballas et al. (2016) + FT
Pan et al. (2016b) + C3D
Yu et al. (2016) + C3D

29.60
29.8
30.75
31.0
32.60

−
−
−
78.80
81.50

−
−
−
66.0
70.40

−
−
−
55.4
60.4

41.92
−
49.0
45.3
49.90

51.67
−
59.37
−
−

Table 2. Video captioning evaluation on MSVD (670 videos).

Method

M

Human
24
(Sigurdsson et al., 2016)

B@1 B@2 B@3 B@4 C

References

62

Bahdanau, Dzmitry, Cho, Kyunghyun, and Bengio,
Yoshua. Neural machine translation by jointly learning
to align and translate. ICLR, 2015.

Sigurdsson et al. (2016) 16 49
17.6 50
Our Model

43

29

20

53

30 18 11 14
31.1 18.8 11.5 16.7

Ballas, Nicolas, Yao, Li, Pal, Chris, and Courville,
Aaron C. Delving deeper into convolutional networks
for learning video representations. In ICLR, 2016.

Table 3. Video captioning evaluation on Charades (1863 videos).
M=M ETEOR, B=BLEU, C=CIDEr. Sigurdsson et al. (2016) results
use the Venugopalan et al. (2015a) model.

Bergstra, James and Bengio, Yoshua. Random search for
hyper-parameter optimization. J. Mach. Learn. Res., 13:
281–305, 2012.
natural language description. Our model utilizes a deep
learning architecture that represents video with an explicit
model of the video’s temporal structure, and jointly models the video description and the temporal video sequence.
This effectively connects the visual video space and the language description space.

Brox, T. and Malik, J. Large displacement optical flow:
Descriptor matching in variational motion estimation.
TPAMI, 33(3):500–513, March 2011. ISSN 0162-8828.
Brox, T., Bruhn, A., Papenberg, N., and Weickert, J. High
accuracy optical flow estimation based on a theory for
warping. In ECCV, 2004.

A memory-based attention mechanism helps guide where
to attend and what to reason about as the description is generated. This allows the model to not only reason efficiently
about local attention, but also to consider the full sequence
of video frames during the generation of each word. Our
experiments confirm that the memory components in our
architecture, most notably from the IAM module, play a
significant role in improving the performance of the entire
network.
Future work should raim to refine the temporal video frame
model, TEM, and explore how to improve performance on
capturing the ideal frames for each description.

Chen, David L. and Dolan, William B. Collecting highly
parallel data for paraphrase evaluation. In ACL, June
2011.
Devlin, Jacob, Cheng, Hao, Fang, Hao, Gupta, Saurabh,
Deng, Li, He, Xiaodong, Zweig, Geoffrey, and Mitchell,
Margaret. Language models for image captioning: The
quirks and what works. In ACL-IJCNLP, pp. 100–105,
July 2015.

7

Donahue, Jeff, Hendricks, Lisa Anne, Guadarrama, Sergio,
Rohrbach, Marcus, Venugopalan, Subhashini, Saenko,

Example Video Frame Sequence

Proposed Model

Ground Truth

A group of people
are dancing

A group of young
children performing
together

Action error

A person is cutting
the vegetable

A woman is cutting
garlic

Action error, Attention error

A man is playing a
guitar

A man is playing the
guitar

A woman is pouring
eggs into a bowl

A woman is pouring
ingredients into a
bowl

A man is playing a
flute

A man is playing a
large flute

A woman is applying
a makeup

A woman is putting
on makeup

Example Video Frame Sequence

Proposed Model

A man is cutting a
gun

A man is washing a
bath

Ground Truth

A guy is shooting a
gun

A dog is on a
trampoline

Object error in Subject position
A cat is eating

Hamsters are eating

Object error
A young girl is
playing the flute

A little girl is talking
on a cordless
telephone

Figure 3. Example captions generated by our model on MSVD test videos.

Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E.
Imagenet classification with deep convolutional neural
networks. In Pereira, F., Burges, C. J. C., Bottou, L., and
Weinberger, K. Q. (eds.), NIPS, pp. 1097–1105. 2012.

Kate, and Darrell, Trevor. Long-term recurrent convolutional networks for visual recognition and description.
In CVPR, 2015.
Fang, Hao, Gupta, Saurabh, Iandola, Forrest, Srivastava, Rupesh K., Deng, Li, Dollar, Piotr, Gao, Jianfeng, He, Xiaodong, Mitchell, Margaret, Platt, John C.,
Lawrence Zitnick, C., and Zweig, Geoffrey. From captions to visual concepts and back. In CVPR, June 2015.

LeCun, Yann, Bottou, Léon, Bengio, Yoshua, and Haffner,
Patrick. Gradient-based learning applied to document
recognition. Proceedings of the IEEE, 86(11):2278–
2324, 1998.
Pan, Pingbo, Xu, Zhongwen, Yang, Yi, Wu, Fei, and
Zhuang, Yueting. Hierarchical recurrent neural encoder
for video representation with application to captioning.
In CVPR, June 2016a.

Graves, Alex and Jaitly, Navdeep. Towards end-to-end
speech recognition with recurrent neural networks. In
ICML-14, pp. 1764–1772, 2014.
Guadarrama, Sergio, Krishnamoorthy, Niveda, Malkarnenkar, Girish, Venugopalan, Subhashini, Mooney, Raymond, Darrell, Trevor, and Saenko, Kate. Youtube2text:
Recognizing and describing arbitrary activities using semantic hierarchies and zero-shot recognition. In ICCV,
pp. 2712–2719, 2013.

Pan, Yingwei, Mei, Tao, Yao, Ting, Li, Houqiang, and Rui,
Yong. Jointly modeling embedding and translation to
bridge video and language. CVPR, 2016b.
Pascanu, Razvan, Mikolov, Tomas, and Bengio, Yoshua.
On the difficulty of training recurrent neural networks.
ICML-13, 28:1310–1318, 2013.

He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun,
Jian. Deep residual learning for image recognition. In
CVPR, June 2016.

Rocktäschel, Tim, Grefenstette, Edward, Hermann,
Karl Moritz, Kociský, Tomás, and Blunsom, Phil. Reasoning about entailment with neural attention. In ICLR,
2016.

Hochreiter, Sepp and Schmidhuber, Jürgen. Long shortterm memory.
Neural Comput., 9(8):1735–1780,
November 1997. ISSN 0899-7667.

Rohrbach, M., Qiu, W., Titov, I., Thater, S., Pinkal, M., and
Schiele, B. Translating video content to natural language
descriptions. In ICCV, pp. 433–440, Dec 2013.

Johnson, Justin, Karpathy, Andrej, and Fei-Fei, Li. Densecap: Fully convolutional localization networks for dense
captioning. In CVPR, 2016.
Kingma, Diederik P. and Ba, Jimmy. Adam: A method for
stochastic optimization. In ICLR, 2015.

8

Sharma, Shikhar, Kiros, Ryan, and Salakhutdinov, Ruslan. Action recognition using visual attention. CoRR,
abs/1511.04119, 2015.

Shin, Andrew, Ohnishi, Katsunori, and Harada, Tatsuya.
Beyond caption to narrative: Video captioning with multiple sentences. ICIP, 2016.

Yao, Li, Torabi, Atousa, Cho, Kyunghyun, Ballas, Nicolas, Pal, Christopher, Larochelle, Hugo, and Courville,
Aaron. Describing videos by exploiting temporal structure. In ICCV, 2015.

Sigurdsson, Gunnar A., Varol, Gül, Wang, Xiaolong,
Farhadi, Ali, Laptev, Ivan, and Gupta, Abhinav. Hollywood in homes: Crowdsourcing data collection for activity understanding. In ECCV, 2016.

Yu, Haonan, Wang, Jiang, Huang, Zhiheng, Yang, Yi, and
Xu, Wei. Video paragraph captioning using hierarchical
recurrent neural networks. In CVPR, June 2016.

Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. CoRR,
abs/1409.1556, 2014.
Sukhbaatar, Sainbayar, Szlam, Arthur, Weston, Jason, and
Fergus, Rob. End-to-end memory networks. NIPS, 2015.
Sutskever, Ilya, Vinyals, Oriol, and Le, Quoc V. Sequence
to sequence learning with neural networks. In NIPS, pp.
3104–3112. 2014.
Tran, Du, Bourdev, Lubomir, Fergus, Rob, Torresani,
Lorenzo, and Paluri, Manohar. Learning spatiotemporal features with 3d convolutional networks. In ICCV,
2015.
Venugopalan, Subhashini, Rohrbach, Marcus, Donahue,
Jeff, Mooney, Raymond, Darrell, Trevor, and Saenko,
Kate. Sequence to sequence – video to text. In ICCV,
2015a.
Venugopalan, Subhashini, Xu, Huijuan, Donahue, Jeff,
Rohrbach, Marcus, Mooney, Raymond, and Saenko,
Kate. Translating videos to natural language using deep
recurrent neural networks. In NAACL HLT, 2015b.
Vinyals, Oriol, Toshev, Alexander, Bengio, Samy, and Erhan, Dumitru. Show and tell: A neural image caption
generator. In CVPR, June 2015.
Weston, Jason, Chopra, Sumit, and Bordes, Antoine. Memory networks. CoRR, abs/1410.3916, 2014.
Xu, Kelvin, Ba, Jimmy, Kiros, Ryan, Cho, Kyunghyun,
Courville, Aaron, Salakhudinov, Ruslan, Zemel, Rich,
and Bengio, Yoshua. Show, attend and tell: Neural image caption generation with visual attention. In ICML15, pp. 2048–2057, 2015a.
Xu, Ran, Xiong, Caiming, Chen, Wei, and Corso, Jason J.
Jointly modeling deep video and compositional text to
bridge vision and language in a unified framework. In
AAAI, 2015b.
Yang, Zhilin, Yuan, Ye, Wu, Yuexin, Salakhutdinov, Ruslan, and Cohen, William W. Encode, review, and decode: Reviewer module for caption generation. CoRR,
abs/1605.07912, 2016.

9

