Task-driven Visual Saliency and Attention-based Visual Question Answering
Yuetan Lin

Zhangyang Pang
Donghui Wang∗
Yueting Zhuang
College of Computer Science, Zhejiang University
Hangzhou, P. R. China

Image

Abstract

Attention Maps

Q1

Visual question answering (VQA) has witnessed great
progress since May, 2015 as a classic problem unifying visual and textual data into a system. Many enlightening VQA
works explore deep into the image and question encodings
and fusing methods, of which attention is the most effective and infusive mechanism. Current attention based methods focus on adequate fusion of visual and textual features,
but lack the attention to where people focus to ask questions about the image. Traditional attention based methods attach a single value to the feature at each spatial location, which losses many useful information. To remedy
these problems, we propose a general method to perform
saliency-like pre-selection on overlapped region features by
the interrelation of bidirectional LSTM (BiLSTM), and use
a novel element-wise multiplication based attention method
to capture more competent correlation information between
visual and textual features. We conduct experiments on the
large-scale COCO-VQA dataset and analyze the effectiveness of our model demonstrated by strong empirical results.

Region
Pre-selection

arXiv:1702.06700v1 [] 22 Feb 2017

{linyuetan,pzy,dhwang,yzhuang}@zju.edu.cn

Q3
Saliency
Weight Map

EWM
Attention

A2

Q2: Is this
person cold?
A2: yes

Q3: What
A3 color are the
skis?
Answer A3: red
Generation

Figure 1. (Best view in color and zoom in.) The flow of our proposed VQA model. Q and A represent question and answer related
to the image. EWM denotes element-wise multiplication operation.

and a big VQA challenge [3] have been proposed so far.
Four commonly used datasets (i.e. DAQUAR [21], COCOQA [24], COCO-VQA [3] and Visual7W [36]) feature different aspects. The common practice to tackle VQA problem is to translate the words as word embeddings and encode the questions using bag-of-word (BoW) or Long Short
Term Memory (LSTM) network, and encode the images using deep convolutional neural networks (CNN). The following important step is to combine the image and question representations through some kind of fusing methods for answer generation, such as concatenation [35, 22, 6], elementwise multiplication [3], parameter prediction layer [23],
episode memory [31], attention mechanism [26, 13, 19],
etc. Current VQA works focus on the fusion of these two
features, yet no one cares about “where we focus” to ask
questions on the image. It is a common practice to treat
the VQA problem as either a generation, classification or a
scoring task, and classification gains more popularity due to
its simplicity and easiness for comparison.
These works treat VQA as a discriminative model, learning the conditional probability of answer given the image
and question. From the generative view, we emulate the behavior that before people ask questions about the given image they first glance at it and find some interesting regions.

1. Introduction
Visual question answering (VQA) comes as a classic task
which combines visual and textual modal data into a unified system. Taking an image and a natural language question about it as input, a VQA system is supposed to output
the corresponding natural language answer. VQA problem
requires image and text understanding, common sense and
knowledge inference. The solution of VQA problem will
be a great progress in approaching the goal of Visual Turing Test, and is also conducive to tasks such as multi-modal
retrieval, image captioning and accessibility facilities.
After the first attempt and introduction of VQA [21],
more than thirty works on VQA have sprung up over the
past one year from May, 2015. Over ten VQA datasets
∗ Corresponding

Q2

A1

Q1: What is
the shed
make of in the
background?
A1: wood

author

1

In terms of a single person, he has unique taste for choosing image regions that interest him. For a large amount of
people, there are statistical region-of-interest (RoI) distributions. These region patterns are task-driven, e.g. the picture in Figure. 1, for VQA task people may focus mostly
on the beds, the chairs, the laptop and the notebook regions (namely the RoI patterns) as captured in the weighted
image, but for image captioning task they pay attention to
more areas including the striped floor. It is very valuable
to intensify the interesting region features and suppress others, and this image preprocessing step provides more accurate visual features to the follow-up steps and is missing
in current VQA works. By analogy with visual saliency
which captures the standing out regions or objects of an image, we propose a region pre-selection mechanism named
task-driven visual saliency which attaches interesting regions (more possibly questioned on) with higher weights.
Taking advantage of the bidirectional LSTM (BiLSTM) [7]
that the output at an arbitrary time step has complete and
sequential information about all time steps before and after it, we compute the weight of interest for each region
feature which is relative to all of them. To the best of our
knowledge, this is the first work that employs and analyzes
BiLSTM in VQA models for task-driven saliency detection,
and this is the first contribution of our work.
As a simple and effective VQA baseline method, [35]
shows that question feature always contributes more to predict the answer than image feature. But image is as equally
important as question for answer generation. It is necessary
to further explore finer-grained image features to achieve
better VQA performance, e.g. attention mechanism [32].
Current attention based models generally use the correlation scores between question and image representations as
weights to perform weighted sum of region features, the resulting visual vector is concatenated to the question vector for final answer generation. The recent “multi-step” attention models (i.e. containing multiple attention layers)
[34, 19] dig deeper into the image understanding and help
achieve better VQA performance than the “regular” attention models. However, the correlation score obtained by
inner product between visual and textual features is essentially the sum of the correlation vector obtained by elementwise multiplication of the two features. Besides, [3] shows
that element-wise multiplication of these features achieves
more accurate results than concatenation of them in the
baseline model. Hence we propose to employ element-wise
multiplication way in the attention mechanism, the fused
features are directly feed forward to a max pooling layer to
get the final fused feature. Together with the saliency-like
region pre-selection operation, this novel attention method
effectively improves VQA performance and is the second
contribution of this work.
The remainder of the paper is organized as follows. We

first briefly review saliency and the attention mechanism.
Then, we elaborate our proposed method. We present experiments of some baseline models and compare with stateof-the-art models and visualize the pre-selection saliency
and attention maps. Finally we summarize our work.

2. Related Work
2.1. Saliency Detection Modeling
Saliency generally comes from contrasts between a pixel
or an object and its surroundings, describing how outstanding it is. It could facilitate learning by focusing the
most pertinent regions. Saliency detection methods mimic
the human attention in psychology, including both bottomup and top-down manners [29]. Typical saliency methods [10, 18] are pixel- or object-oriented, which are not appropriate for VQA due to center bias and are difficulty in
collecting large scale eye tracking data.
We think task-driven saliency on image features could
be conductive to solving VQA problem. What inspires us
is that BiLSTM used in saliency detection has achieved
good results on text and video tasks. In sentiment classification tasks, [16] assigns saliency scores to words related
to sentiment for visualizing and understanding the effects of
BiLSTM in textual sentence. While in video highlight detection, [33] uses a recurrent auto-encoder configured with
BiLSTM cells and extracts video highlight segments effectively. BiLSTM has demonstrated its effectiveness in
saliency detection, but to the best of our knowledge it has
not been used in visual saliency for VQA task.

2.2. Attention in VQA Models
Visual attention mechanism has drawn great interest in
VQA [34, 36, 26] and gained performance improvement
from traditional methods using holistic image features. Attention mechanism is typically the weighted sum of the
image region features at each spatial location, where the
weights describe the correlation and are implemented as the
inner products of the question and image features. It explores finer-grained visual features and mimics the behavior
that people attend to different areas according to the questions. Focusing on “knowing where to look” for multiplechoice VQA tasks, [26] uses 99 detected object regions plus
a holistic image feature to make correlation with the question encoding, and uses the correlation scores as weights to
fuse the features. [34] uses the last pooling layer features
(512 × 14 × 14) of VGG-19 [27] as image region partitions, and adopts two-layer attention to obtain more effective fused features for complex questions. [2] proposes an
ingenious idea to use assembled network modules according to the parsed questions, and achieves multi-step transforming attention by specific rules.
However, these attention methods use correlation score

(i.e. inner product between visual and textual feature) for
each location, which is the sum of the correlation vector
representation (i.e. element-wise multiplication between
them). Besides, the concatenation of image and question
features is less accurate than the element-wise multiplication vector of them shown in the baseline model [3]. Moreover, there are many answers derived from non-object and
background regions, e.g. questions about scenes, hence it is
not fit for the object detection based attention methods.

3. Proposed Method
Compared with image captioning which generates general descriptions about an image, VQA focuses on specific
image regions depending on the question. On the one hand,
these regions include non-object and background contents
which are hard for object detection based VQA methods.
On the other hand, although people may ask questions at
any areas of a given image, there are always some region
patterns that attract more questions. On the whole, there are
statistical region-of-interest (RoI) patterns which represent
human-interested areas that are important for later VQA
task. We propose a saliency-like region pre-selection and
attention-based VQA framework illustrated in Figure. 2.
The VQA is regarded as a classification task, which is simple and easy to transform to a generating or scoring model.

3.1. Model
In this section, we elaborate our model consisting of four
parts: (a) image feature pre-selection part which models the
tendency where people focus to ask questions, (b) question
encoding part which encodes the question words as a condensed semantic embedding, (c) attention-based feature fusion part performs second selection on image features and
(d) answer generation part which gives the answer output.
3.1.1

Image Feature Pre-selection

As described above, current object detection based VQA
methods may not be qualified and the answers may not
be derived from these specific object regions in images,
for example, when asked “Where is the bird/cat?”, the answers “fence/sink” are not contained in ILSVRC [25] (200
categories) and Pascal VOC [5] (20 categories) detection
classes. Thus we use a more general pattern detector.
In addition, from the generative perspective, we pay attention to where people focus to ask questions. General visual saliency provides analogous useful information of noticeable objects or areas which outstand the surroundings,
but it is not the only case for VQA task. Current attention
mechanism relates the question to the focusing location. As
more samples are available, we could yield the region patterns that attract more questions by statistics. From the statistical behavior of large amounts of workers on Amazon

Mechanical Turk (AMT) who have labeled the questions,
we model the region-of-interest patterns that could attract
more questions.
We propose to perform saliency-like pre-selection operation to alleviate the problems and model the RoI patterns.
The image is first divided into g × g grids as illustrated in
Figure. 2. Taking m×m grids as a region, with
 as the
 s grids
+ 1. We
stride, we obtain n × n regions, where n = g−m
s
then feed the regions to a pre-trained ResNet [8] deep convolutional neural network to produce n×n×dI -dimensional
region features, where dI is the dimension of feature from
the layer before the last fully-connected layer.
Since the neighboring overlapped regions share some visual contents, the corresponding features are related but focusing on different semantic information. We regard the
sequence of regions as the result of eye movement when
glancing at the image, and these regions are selectively allocated different degrees of interest. Specifically, the LSTM
is a special kind of recurrent neural network (RNN), capable of learning long-term dependencies via the memory cell
and the update gates, which endows itself with the ability
to retain information of previous time-steps (i.e. the previous region sequence in this case). The update rules of the
LSTM at time step t are as follows:
it = σ(W (i) xt + U (i) ht−1 + b(i) ),
ft = σ(W

(f )

xt + U

(f )

ot = σ(W

(o)

xt + U

(o)

ht−1 + b

ht−1 + b

(f )

(o)

(1)

),

(2)

),

(3)

ut = tanh(W (u) xt + U (u) ht−1 + b(u) ),

(4)

ct = ut

it + ct−1

(5)

ht = ot

tanh(ct ),

ft ,

(6)

where i, f, o denote the input, forget and output gates,
x, c, h are the input region feature, memory cell and hidden
unit output, and W, U, b are the parameters to be trained.
We activate the gates by the sigmoid nonlinearity σ(x) =
1/(1 + e−x ) and the cell contents by the hyperbolic tangent
tanh(x) = (ex − e−x )/(ex + e−x ). The gates control the
information in the memory cell to be retained or forgotten
through element-wise multiplication .
Inspired by the information completeness and high performance of BiLSTM, we encode the region features in two
directions using BiLSTM and obtain a scalar output per region. The output of the BiLSTM is the summation of the
forward and backward LSTM outputs at this region loca(f )
(b)
tion: ht = ht + hn−t+1 , where n is the number of re(f )
(b)
gions, ht , hn−t+1 are computed using Eq. 6. Hence, the
output at each location is influenced by the region features
before and after it, which embodies the correlation among
these regions. Note that, although the DMN+ work [31]
uses similar bi-directional gated recurrent units (BiGRU) in
the visual input module, their purpose is to produce input

CNN
BiLSTM

...

...

...

...

CNN

Softmax Layer

BiLSTM

Max Pooling Layer

CNN

tie

BiLSTM

Image Feature
Pre-selection

CNN

Attention-based
Feature Fusion

BiLSTM

Answer
Generation

Question Encoding
What is the green object below the man’s head?

LSTM

Figure 2. (Best view inNcolor J
and zoom in.) Frameworks of our proposed VQA model. Bars with frame lines denote vectors and small
circles denote scalars.
and
represent multiplication of scalar and vector and element-wise multiplication of two vectors, respectively.

facts which contain global information. Besides, their BiGRU takes the features embedded to the textual space as
inputs. In contrast, the BiLSTM used in our model takes
directly visual CNN features as input, and the main purpose
is to output weights for region feature selection.
The output values of the BiLSTM are normalized
through a softmax layer, and the resulting weights are multiplied by the region features. We treat the weights as degree of interest which are trained by error back-propagation
of the final class cross entropy losses, and higher weights
embody that the corresponding region patterns will attract
more questions, in other words, these region patterns may
get higher attention values in the latter interaction with
question embeddings in a statistical way.
3.1.2

Question Encoding

Question can be encoded using various kinds of natural language processing (NLP) methods, such as BoW, LSTM,
CNN [20, 34], gated recurrent units (GRU) [4], skipthought vectors [14], or it can be parsed by Stanford Parser
[15], etc. Since question BoW encodings already dominate
the contribution to answer generation compared with the
image features [35], we simply encode the question word
as word2vec embedding, and use LSTM to encode the questions to match the pre-selected region features. To encode
more abstract and higher-level information and achieve better performance, a deeper LSTM [3, 12] for question encoding is adopted in our model.
The question encoding LSTM in our model has l hidden
layers with r hidden units per layer, and the question representation is the last output and the cell units of the LSTM,

and the dimension is dQ = 2 × l × r. The resulting condensed feature vector encodes the semantic and syntactic
information of the question.
3.1.3

Attention-based Feature Fusion

According to the statistic image-question-answer (IQA)
training triples, the image feature pre-selection has attached
the regions with different prior weights, generating more
meaningful region features. But different questions may focus on different aspects of the visual content. It is necessary
to use attention mechanism to second select regions by the
question for more effective features.
We propose a novel attention method, which takes the
element-wise multiplication vector as correlation between
image and question features at each spatial location. Specifically, given the pre-selected region features and question
embedding, we map the visual and textual features into a
common space of dC dimension and perform element-wise
multiplication between them. The n × n × dC -dimensional
fused features contain visual and textual information, and
higher responses indicate more correlative features. In
traditional attention models, the correlation score (scalar)
achieved by inner product between the mapped visual and
textual features per region, is essentially the sum of elements in our fused feature. This novel attention method
has two noticeable advantages against traditional attention,
i.e. information richer correlation vector versus correlation scalar, more effective element-wise multiplication vector versus the concatenated vector of the visual and textual
features.
Since higher responses in the fused features indicate

more correlative visual and textual features, and the question may only focus on one or two regions. We choose to
apply max pooling operation on the intermediate fused features to pick out the maximum responses. The produced
dC -dimensional fused feature is then fed to the final answer
generation part. Compared to the sum/average operation in
traditional attention models, the max operation highlights
the responses of the final fused feature from every spatial
location.
3.1.4

Answer Generation

Taking the VQA problem as a classification task is simple to
be implemented and evaluated, and it is easy to be extended
to generation or multiple choice tasks through a network
surgery using the fused feature in the previous step. We use
a linear layer and a softmax layer to map from the fused
feature to the answer candidates, of which the entries are
the top-1000 answers from the training data.
Considering multiple choice VQA problems, e.g. Visual7W [36] telling questions and COCO-VQA [3] multiple choice tasks, our model is adaptive to be extended by
concatenating the question and answer vectors before fusion with visual features or by using bilinear model between
the final fused feature and answer feature [26, 11], which
is a possible future work. Meanwhile, in view of generation VQA problem, we can train an LSTM taking the fused
feature as input to obtain answer word lists, phrases or sentences [22, 6].

3.2. Training
Our framework is trained end-to-end using backpropagation, while the feature extraction part using ResNet
is kept fixed to speed up training and avoid the noisy gradients back-propagated from the LSTM as elaborated in [6].
RMSprop algorithm is employed with low initial learning
rate of 3e-4 which is proved important to prevent the softmax from spiking too early and prevent the visual features
from dominating too early [26]. Due to simplicity and
proved similar performance as pre-trained word embedding
parameters, we initialize the parameters of the network with
random numbers. We randomly sample 500 IQA triples per
iteration.

4. Experiments
In this section, we describe the implementation details
and evaluate our model (SalAtt) on the large-scale COCOVQA dataset. Besides, we visualize and analyze the role of
pre-selection and the novel attention method.

4.1. Implementation Details
In our experiment, the input images are first scaled to
448 × 448 × 3 pixels before we apply 4 × 4 grids on them.

We obtain 3 × 3 regions by employing 2 × 2 grids (i.e.
224×224×3 pixels) as a region with stride 1 grid. Then we
extract the 2048-D feature per region from the layer before
the last fully-connected layer of ResNet. The dimension
of word embedding is 200, and the weights of the embedding are initialized randomly from a uniform distribution on
[−0.08, 0.08) due to similar performance as the pre-trained
one. The pre-selection BiLSTM for region features has 1
layer and the size is 1, and the LSTM for question uses 2
layers and 512 hidden units per layer. The common space
of visual and textual features is 1024-dimensional. We use
dropout [28] after all convolutional and linear layers. The
non-linear function is hyperbolic tangent.
The training procedure is early stopped when there is
no accuracy increase in validation set for 5, 000 iterations
where we evaluate every 1, 000 iterations. It takes around
18 hours to train our model on a single NVIDIA Tesla K40
GPU for about 91, 000 iterations. And for evaluation, each
sample needs less than 0.5 millisecond.

4.2. Datasets
The COCO-VQA dataset [3] is the largest among the
commonly used VQA datasets, which contains two tasks
(i.e. multiple-choice task and open-ended task) on two image datasets (i.e. real image MSCOCO dataset [17] and abstract scene dataset). We follow the common practice to
evaluate models on two tasks on the real image dataset,
which includes 248,349 training questions, 121,512 validation questions and 244,302 testing questions. There are
many types of questions which require image and question
understanding, commonsense knowledge, knowledge inference and even external knowledge. The answers are roughly
divided into 3 types, i.e. “yes/no”, “number” and “other”.
To evaluate the results, each answer is compared with
10 human-labeled answers, the accuracy is computed via
answers
, 1),
this metric: min( #consistent human−labeled
3
i.e. the accuracy is 100% if the predicted answer is consistent with at least 3 human-labeled answers. The COCOVQA dataset provide human-labeled answers for the training and validation sets, and the results of testing set can only
be tested on the evaluation server. The whole testing set is
named test-standard and can be evaluated once per day and
5 times in total, and a smaller development set named testdev can be tested 10 times per day and 9999 times in total.
In short, the COCO-VQA dataset is large and hard enough
for evaluating models and hence we choose to evaluate our
model on it.

4.3. Compared Models
We compare our propose model (SalAtt) with some
function-disabled models listed below, to prove the effectiveness of the region pre-selection via BiLSTM and the
novel attention method.

Model

holistic

TraAtt

RegAtt

ConAtt

SalAtt

VGG19
ResNet

59.52
60.30

58.39
59.07

59.76
60.45

59.24
60.25

59.86
60.62

Table 1. Comparison of different models using two CNN features on COCO-VQA validation set, where SalAtt is our proposed
model. (accuracy in %)

• holistic: The baseline model which maps the holistic
image feature and LSTM-encoded question feature to
a common space and perform element-wise multiplication between them.
• TraAtt: The traditional attention model, implementation of WTL model [26] using the same 3 × 3 regions
in SalAtt model.
• RegAtt: The region attention model which employs
our novel attention method, same as the SalAtt model
but without region pre-selection.
• ConAtt: The convolutional region pre-selection attention model which replaces the BiLSTM in SalAtt
model with a weight-sharing linear mapping, implemented by a convolutional layer.
Besides, we also compare our SalAtt model with the
popular baseline models i.e. iBOWIMG [35], VQA [3], and
the state-of-the-art attention-based models i.e. WTL [26],
NMN [2], SAN [34], AMA [30], FDA [9], D-NMN [1],
DMN+ [31] on two tasks of COCO-VQA.

4.4. Results and Analysis
4.4.1

Effectiveness of Proposed Functions

We train the function-disabled models on COCO-VQA
training set and show the accuracies on validation set in
Table. 1. From the columns, we can see that: (1) holistic
is better than TraAtt, proving the effectiveness of elementwise multiplication feature fusion compared with concatenation of features. (2) RegAtt is better than holistic, indicating our novel attention method indeed enriches the visual features and improves the performance. (3) SalAtt is
better than RegAtt, demonstrating the strength of our region pre-selection mechanism. (4) ConAtt is worse than
SalAtt, showing that BiLSTM is important for the region
pre-selection part. From each row, we find the consistent
improvement by the ResNet features, showing the importance of good CNN features to VQA.
4.4.2

Quantitative Results on Evaluation Server

We summarize the accuracies on test-dev in Table. 2 and
the test-standard results in Table. 3. Our results are comparative or higher than the attention based methods, especially on multiple-choice tasks. The results on answer type

“other”, which includes object and scene type questions,
demonstrate the competence of our model in RoI detection.
Note that, we only apply the proposed region preselection mechanism to the basic VQA model [3], it can
be embedded into any other attention-based models to improve their performance. Due to computation and training time, we use only 3 × 3 regions compared with other
attention-based methods (e.g. 100 or 14 × 14 region features). Through observation, we find that many small objects could not be split by the 3×3 regions, which is adverse
to the counting questions and could be further improved and
is a possible future work.
4.4.3

Visualization of Intermediate Results

We illustrate three groups of samples produced by our
model in Figure. 3. Each group contains four figures, from
left to right and from top to bottom, they are respectively
the original image, pre-selection weights on the image, and
two attention maps for different questions with the corresponding questions (Q), ground truth answers (A) and the
predicted answers (P) shown below them. And the number
in the parentheses means the amount for this human-labeled
answer entry. The weights are normalized to have minimum
0 and maximum 1 for visualization enhancement, i.e. the
weight in the dark region may not necessarily be 0.
Take the first sample for example, the pre-selection operation gives high weight to the boy’s head region which
may be interesting to people and attract more questions (e.g.
questions containing the word “boy”). For the question “Is
the boy dressed for the weather?”, the attention map focuses
on the boy, his clothes and the surrounding regions to get a
positive answer. While for question “What is the boy doing?”, it attends the boy and the snowboard, thus giving
answer “snowboarding”. The third sample gives inaccurate but explainable answers, i.e. the birds may live in the
park/zoo and come for food provided by the tourist so it may
not be classified into pets, and the left hand of the woman
holds indeed a phone while the human-labeled answers focus on the right hand.

5. Conclusion
In this work, we propose a general VQA solution which
integrates region pre-selection and a novel attention method
to capture generic class region and richer fused feature representation. These two procedures are independent, meanwhile they both contribute to better VQA performance. Although the model is simple, it achieves comparative or
higher empirical results than state-of-the-art models.
Possible future works include adopting finer-grained
grids which capture more precise regions, employing
stacked attention layers for multi-step reasoning and more
accurate answer location, and applying the general pre-

Open Ended

Multiple Choice

All

Yes/No

Number

Other

All

Yes/No

Number

Other

iBOWIMG
VQA
WTL
NMN
SAN
AMA
FDA
D-NMN
DMN+

55.72
57.75
58.60
58.70
59.17
59.24
59.40
60.30

76.55
80.50
81.20
79.30
81.01
81.14
81.10
80.50

35.03
36.77
38.00
36.60
38.42
36.16
38.60
36.80

42.62
43.08
44.00
46.10
45.23
45.77
45.50
48.30

61.68
62.70
62.44
64.01
-

76.68
80.52
77.62
81.50
-

37.05
38.22
34.28
39.00
-

54.44
53.01
55.84
54.72
-

Ours

58.97

79.39

36.29

46.67

63.69

79.42

38.12

56.02

Table 2. Accuracies for the open-ended and multiple-choice tasks on VQA test-dev evaluated on the VQA Server.

Open Ended

Multiple Choice

All

Yes/No

Number

Other

All

Yes/No

Number

Other

iBOWIMG
VQA
NMN
SAN
D-NMN
AMA
WTL
FDA
DMN+

55.89
58.16
58.70
58.90
59.44
59.44
59.54
60.36

76.76
80.56
80.98
81.07
81.34
80.43

34.98
36.53
37.48
37.12
35.67
36.82

42.62
43.73
45.81
45.83
46.10
48.33

61.97
63.09
63.53
64.18
-

76.86
80.59
78.08
81.25
-

37.30
37.70
34.26
38.30
-

54.60
53.64
57.43
55.20
-

Ours

59.26

79.42

36.35

47.00

63.99

79.45

37.93

56.42

Table 3. Results for both tasks on VQA test-standard.

Q: Is the boy
dressed for the
weather?
A: yes(10)
P: yes

Q: What is the boy
doing?
A: snowboarding(9);
skiing(1)
P: snowboarding

Q: What season is
this picture taken?
A: fall(6); winter(2);
autumn(1);
spring(1)
P: fall

Q: How many long
benches are there?
A: 4(4); 3(3); 15(2);
about 10(1)
P: 4

Q: Are those
birds pets?
A: yes(7);
no(3)
P: no

Q: What is the woman
holding?
A: birds(6); feeding
birds(1); parrots(1);
parrots and bird
seed(1); cup(1)
P: phone

Figure 3. (Best view in color and zoom in.) Samples generated by our model. The weights are normalized for visualization enhancement,
i.e. the weight in the dark region may not necessarily be 0. Q, A, P represent question, ground truth answer, predicted answer, respectively.
Green denotes true prediction and red false.

selection method to other attention-based VQA models.
The pre-selection mechanism is valuable and applicable to
similar task, such as image captioning.

References
[1] J. Andreas, M. Rohrbach, T. Darrell, and D. Klein. Learning to compose neural networks for question answering. In
NAACL, 2016. 6
[2] J. Andreas, M. Rohrbach, T. Darrell, and D. Klein. Neural
module networks. In CVPR, pages 39–48, 2016. 2, 6
[3] S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra,
C. Lawrence Zitnick, and D. Parikh. Vqa: Visual question
answering. In ICCV, pages 2425–2433, 2015. 1, 2, 3, 4, 5, 6
[4] K. Cho, B. Van Merriënboer, D. Bahdanau, and Y. Bengio.
On the properties of neural machine translation: Encoderdecoder approaches. In Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation (SSST-8), 2014.
4
[5] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and
A. Zisserman. The pascal visual object classes (voc) challenge. IJCV, 88(2):303–338, 2010. 3
[6] H. Gao, J. Mao, J. Zhou, Z. Huang, L. Wang, and W. Xu.
Are you talking to a machine? dataset and methods for multilingual image question. In NIPS, pages 2296–2304, 2015.
1, 5
[7] A. Graves and J. Schmidhuber. Framewise phoneme classification with bidirectional lstm and other neural network
architectures. Neural Networks, 18(5):602–610, 2005. 2
[8] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning
for image recognition. In CVPR, June 2016. 3
[9] I. Ilievski, S. Yan, and J. Feng. A focused dynamic attention
model for visual question answering. In ECCV, 2016. 6
[10] L. Itti, C. Koch, E. Niebur, et al. A model of saliencybased visual attention for rapid scene analysis. PAMI,
20(11):1254–1259, 1998. 2
[11] A. Jabri, A. Joulin, and L. van der Maaten. Revisiting visual question answering baselines. arXiv preprint
arXiv:1606.08390, 2016. 5
[12] A. Karpathy, J. Johnson, and L. Fei-Fei. Visualizing and
understanding recurrent networks. ICLR Workshop, 2016. 4
[13] J.-H. Kim, S.-W. Lee, D.-H. Kwak, M.-O. Heo, J. Kim, J.W. Ha, and B.-T. Zhang. Multimodal residual learning for
visual qa. In NIPS, 2016. 1
[14] R. Kiros, Y. Zhu, R. R. Salakhutdinov, R. Zemel, R. Urtasun,
A. Torralba, and S. Fidler. Skip-thought vectors. In NIPS,
pages 3294–3302, 2015. 4
[15] D. Klein and C. D. Manning. Accurate unlexicalized parsing. In ACL, pages 423–430. Association for Computational
Linguistics, 2003. 4
[16] J. Li, X. Chen, E. Hovy, and D. Jurafsky. Visualizing and
understanding neural models in nlp. NAACL, 2016. 2
[17] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In ECCV, pages 740–755. Springer,
2014. 5

[18] T. Liu, Z. Yuan, J. Sun, J. Wang, N. Zheng, X. Tang, and
H.-Y. Shum. Learning to detect a salient object. PAMI,
33(2):353–367, 2011. 2
[19] J. Lu, J. Yang, D. Batra, and D. Parikh. Hierarchical
question-image co-attention for visual question answering.
In NIPS, 2016. 1, 2
[20] L. Ma, Z. Lu, and H. Li. Learning to answer questions from
image using convolutional neural network. In AAAI, 2016. 4
[21] M. Malinowski and M. Fritz. A multi-world approach to
question answering about real-world scenes based on uncertain input. In NIPS, pages 1682–1690, 2014. 1
[22] M. Malinowski, M. Rohrbach, and M. Fritz. Ask your neurons: A neural-based approach to answering questions about
images. In ICCV, pages 1–9, 2015. 1, 5
[23] H. Noh, P. H. Seo, and B. Han. Image question answering
using convolutional neural network with dynamic parameter
prediction. In CVPR, June 2016. 1
[24] M. Ren, R. Kiros, and R. Zemel. Exploring models and data
for image question answering. In NIPS, pages 2953–2961,
2015. 1
[25] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh,
S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein,
et al. Imagenet large scale visual recognition challenge.
IJCV, 115(3):211–252, 2015. 3
[26] K. J. Shih, S. Singh, and D. Hoiem. Where to look: Focus
regions for visual question answering. In CVPR, June 2016.
1, 2, 5, 6
[27] K. Simonyan and A. Zisserman. Very deep convolutional
networks for large-scale image recognition. ICLR, 2015. 2
[28] N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and
R. Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning
Research, 15(1):1929–1958, 2014. 5
[29] P. Van De Laar, T. Heskes, and S. Gielen. Task-dependent
learning of attention. Neural Networks, 10(6):981–992,
1997. 2
[30] Q. Wu, P. Wang, C. Shen, A. v. d. Hengel, and A. Dick. Ask
me anything: Free-form visual question answering based on
knowledge from external sources. In CVPR, June 2016. 6
[31] C. Xiong, S. Merity, and R. Socher. Dynamic memory networks for visual and textual question answering. In ICML,
2016. 1, 3, 6
[32] K. Xu, J. Ba, R. Kiros, A. Courville, R. Salakhutdinov,
R. Zemel, and Y. Bengio. Show, attend and tell: Neural
image caption generation with visual attention. In ICML,
volume 37, pages 2048–2057, 2015. 2
[33] H. Yang, B. Wang, S. Lin, D. Wipf, M. Guo, and B. Guo. Unsupervised extraction of video highlights via robust recurrent
auto-encoders. In ICCV, pages 4633–4641, 2015. 2
[34] Z. Yang, X. He, J. Gao, L. Deng, and A. Smola. Stacked
attention networks for image question answering. In CVPR,
June 2016. 2, 4, 6
[35] B. Zhou, Y. Tian, S. Sukhbaatar, A. Szlam, and R. Fergus. Simple baseline for visual question answering. arXiv
preprint arXiv:1512.02167, 2015. 1, 2, 4, 6
[36] Y. Zhu, O. Groth, M. Bernstein, and L. Fei-Fei. Visual7w:
Grounded question answering in images. In CVPR, June
2016. 1, 2, 5

