Dual Recurrent Attention Units for Visual Question Answering
Ahmed Osman 1,2 , Wojciech Samek 1
1
Fraunhofer Heinrich Hertz Institute, Berlin, Germany
2
University of Freiburg, Freiburg, Germany

arXiv:1802.00209v1 [cs.AI] 1 Feb 2018

{ahmed.osman, wojciech.samek}@hhi.fraunhofer.de

Abstract
We propose an architecture for VQA which utilizes recurrent layers to generate visual and textual attention. The
memory characteristic of the proposed recurrent attention
units offers a rich joint embedding of visual and textual features and enables the model to reason relations between
several parts of the image and question. Our single model
outperforms the first place winner on the VQA 1.0 dataset,
performs within margin to the current state-of-the-art ensemble model. We also experiment with replacing attention
mechanisms in other state-of-the-art models with our implementation and show increased accuracy. In both cases,
our recurrent attention mechanism improves performance
in tasks requiring sequential or relational reasoning on the
VQA dataset.

Figure 1. Diagram of the DRAU network.

tial manner which recurrent layers are better suited due to
their ability to capture relevant information over an input
sequence.
In this paper, we propose a RNN-based joint representation to generate visual and textual attention. We argue
that embedding a RNN in the joint representation helps the
model process information in a sequential manner and determine what is relevant to solve the task. We refer to the
combination of RNN embedding and attention as Recurrent
Textual Attention Unit (RTAU) and Recurrent Visual Attention Unit (RVAU) respective of their purpose. Furthermore,
we employ these units in a fairly simple network, referred
to as Dual Recurrent Attention Units (DRAU) network, and
show improved results over several baselines. Finally, we
enhance state-of-the-art models by replacing the model’s
default attention mechanism with RVAU.
Our main contributions are the following:

1. Introduction
Although convolutional neural networks (CNNs) and recurrent neural networks (RNNs) have been successfully
applied to various image and natural language processing
tasks (cf. [1, 2, 3, 4]), these breakthroughs only slowly
translate to multimodal tasks such as visual question answering (VQA) where the model needs to create a joint understanding of the image and question. Such multimodal
tasks require joint visual and textual representations.
Since global features can hardly answer questions about
certain local parts of the input, attention mechanisms have
been extensively used in VQA recently [5, 6, 7, 8, 9, 10,
11, 12]. It attempts to make the model predict based on
spatial or lingual context. However, most attention mechanisms used in VQA models are rather simple, consisting
of two convolutional layers followed by a softmax to generate the attention weights which are summed over the image features. These shallow attention mechanisms may fail
to select the relevant information from the joint representation of the question and image. Creating attention for complex questions, particularly sequential or relational reasoning questions, requires processing information in a sequen-

• We introduce a novel approach to generate soft attention. To the best of our knowledge, this is the first attempt to generate attention maps using recurrent neural
networks. We provide quantitative and qualitative results showing performance improvements over the de1

fault attention used in most VQA models.
• Our attention modules are modular, thus, they can substitute existing attention mechanisms in most models
fairly easily. We show that state-of-the-art models with
RVAU “plugged-in” perform consistently better than
their vanilla counterparts.

2. Related Work
This section discusses common methods that have been
explored in the past for VQA.
Bilinear representations Fukui et al. [7] use compact bilinear pooling to attend over the image features and combine it with the language representation. The basic concept
behind compact bilinear pooling is approximating the outer
product by randomly projecting the embeddings to a higher
dimensional space using Count Sketch projection [13] and
then exploiting Fast Fourier Transforms to compute an efficient convolution. An ensemble model using MCB won first
place in VQA (1.0) 2016 challenge. Kim et al. [5] argues
that compact bilinear pooling is still expensive to compute
and shows that it can be replaced by element-wise product (Hadamard product) and a linear mapping (i.e. fullyconnected layer) which gives a lower dimensional representation and also improves the model accuracy. Recently,
Ben-younes et al. [14] proposed using Tucker decomposition [15] with a low-rank matrix constraint as a bilinear
representation. They propose this fusion scheme in an architecture they refer to as MUTAN which as of this writing
is the current state-of-the-art on the VQA 1.0 dataset.
Attention-based Closely related to our work, Lu et al. [9]
were the first to feature a co-attention mechanism that applies attention to both the question and image. Nam et al.
[6] use a Dual Attention Network (DAN) that employs attention on both text and visual features iteratively to predict
the result. The goal behind this is to allow the image and
question attentions to iteratively guide each other in a synergistic manner.
RNNs for VQA Using recurrent neural networks (RNNs)
for VQA has been explored in the past. Xiong et al. [16]
build upon the dynamic memory network from Kumar and
Varaiya [17] and proposes DMN+. DMN+ uses episodic
modules which contain attention-based Gated Recurrent
Units (GRUs). Note that this is not the same as what we
propose; Xiong et al. generate soft attention using convolutional layers and then uses it to substitute the update gate of
the GRU. In contrast, our approach uses the recurrent layers
to generate the attention. Noh and Han [8] propose recurrent answering units in which each unit is a complete module that can answer a question about an image. They use

joint loss minimization to train the units. However during
testing, they use the first answering unit which was trained
from other units through backpropagation.
Notable mentions Kazemi and Elqursh [18] show that a
simple model can get state-of-the-art results with proper
training parameters. Wu et al. [19] construct a textual representation of the semantic content of an image and merges
it with textual information sourced from a knowledge base.
Ray et al. [20] introduce a task of identifying relevant questions for VQA. Kim et al. [21] apply residual learning techniques to VQA and propose a novel attention image attention visualization method using backpropagation.

3. Dual Recurrent Attention in VQA
We propose our method in this section. Figure 1 illustrates the flow of information in the DRAU model. Given
an image and question, we create the input representations
v and q. Next, these features are combined by 1 × 1 convolutions into two separate branches. Then, the branches
are passed to an RTAU and RVAU. Finally, the branches are
combined using a fusion operation and fed to the final classifier. The full architecture of the network is depicted in
Figure 2.

3.1. Input Representation
Image representation We use the 152-layer “ResNet”
pretrained CNN from He et al. [1] to extract image features.
Similar to [7, 6], we resize the images to 448 × 448 and
extract the last layer before the final pooling layer (res5c)
with size 2048 × 14 × 14. Finally, we use l2 normalization on all dimensions. Recently, Anderson et al. [22] have
shown that object-level features can provide a significant
performance uplift compared to global-level features from
pretrained CNNs. Therefore, we experiment with replacing
the ResNet features with FRCNN [23] features with a fixed
number of proposals per image (K = 36).
Question representation We use a fairly similar representation as [7]. In short, the question is tokenized and encoded using an embedding layer followed by a tanh activation. We also exploit pretrained GloVe vectors [24] and
concatenate them with the output of the embedding layer.
The concatenated vector is fed to a two-layer unidirectional
LSTM that contains 1024 hidden states each. In contrast
to Fukui et al., we use all the hidden states of both LSTMs
rather than concatenating the final states to represent the final question representation.

3.2. 1 × 1 Convolution and PReLU
We apply multiple 1 × 1 convolution layers in the network for mainly two reasons. First, they learn weights from

Figure 2. The proposed network. ⊕ denotes concatenation.

the image and question representations in the early layers.
This is important especially for the image representation,
since it was originally trained for a different task. Second,
they are used to generate a common representation size.
To obtain a joint representation, we apply 1 × 1 convolutions followed by PReLU activations [1] on both the image
and question representations. Through empirical evidence,
PReLU activations were found to reduce training time significantly and improve performance compared to ReLU and
tanh activations. We provide these results in Section 4.

3.3. Recurrent Attention Units
The result from the above-mentioned layers is concatenated and fed to two separate recurrent attention units
(RAU). Each RAU starts with another 1 × 1 convolution
and PReLU activation:

ca = PReLU Wa x
(1)
where Wa is the 1 × 1 convolution weights, x is the input to
the RAU, and ca is the output of the first PReLU.
Furthermore, we feed the previous output into an unidirectional LSTM:

ha,n = LSTM ca,n
(2)

Next, we use the attention weights to compute a
weighted average of the image and question features.
atta,n =

N
X

Watt,n fn

(4)

n=1

where fn is the input representation and atta,n is the attention applied on the input. Finally, the attention maps are fed
into a fully-connected layer followed by a PReLU activation. Figure 3 illustrates the structure of a RAU.

yatt,n = PReLU Wout atta,n
(5)
where Wout is a weight vector of the fully connected layer
and yatt,n is the output of each RAU.

3.4. Reasoning layer
A fusion operation is used to merge the textual and visual
branches. For DRAU, we experiment with using elementwise multiplication (Hadamard product) and MCB [7, 25].
The result of the fusion is given to a many-class classifier
using the top 3000 frequent answers. We use a single-layer
softmax with cross-entropy loss. This can be written as:



Pa = softmax fusion op ytext , yvis Wans
(6)

where ha,n is the hidden state at time n.
To generate the attention weights, we feed all the hidden
states of the previous LSTM to a 1 × 1 convolution layer
followed by a softmax function. The 1×1 convolution layer
could be interpreted as the number of glimpses the model
sees.


Watt,n = softmax PReLU Wg ha,n
(3)

where ytext and yvis are the outputs of the RAUs, Wans
represents the weights of the multi-way classifier, and Pa is
the probability of the top 3000 frequent answers.
The final answer â is chosen according to the following:

where Wg is the glimpses’ weights and Watt,n is the attention weight vector.

Experiments are performed on the VQA 1.0 and 2.0
datasets [26, 27]. These datasets use images from the

â = argmax Pa

(7)

4. Experiments and Results

VQA 1.0 Validation Split
Open Ended Task
Baselines
Y/N Num. Other
Language only
Simple MCB
Joint LSTM

Figure 3. Recurrent Attention Unit.

MS-COCO dataset [28] and generate questions and labels
(10 labels per question) using Amazon’s Mechanical Turk
(AMT). Compared to VQA 1.0, VQA 2.0 adds more imagequestion pairs to balance the language prior present in the
VQA 1.0 dataset. The ground truth answers in the VQA
dataset are evaluated using human consensus.

P
a is in human annotation
,1
(8)
Acc(a) = min
3
We evaluate our results on the validation, test-dev, teststd splits of each dataset. Models evaluated on the validation set use train and Visual Genome for training. For
the other splits, we include the validation set in the training
data. However, the models using FRCNN features do not
use data augmentation with Visual Genome.
To train our model, we use Adam [29] for optimization
with β1 = 0.9, β2 = 0.999, and an initial learning rate
of  = 7 × 10−4 . The final model is trained with a small
batch size of 32 for 400K iterations. We did not fully explore tuning the batch size which explains the relatively
high number of training iterations. Dropout (p = 0.3) is
applied after each LSTM and after the fusion operation. All
weights are initialized as described in [30] except LSTM
layers which use an uniform weight distribution. The pretrained ResNet was fixed during training due to the massive
computational overhead of fine-tuning the network for the
VQA task. While VQA datasets provide 10 answers per
image-question pair, we sample one answer randomly for
each training iteration.

4.1. VQA 1.0 Experiments
During early experiments, the VQA 2.0 dataset was not
yet released. Thus, the baselines and early models were
evaluated on the VQA 1.0 dataset. While building the final model, several parameters were changed, mainly, the
learning rate, activation functions, dropout value, and other
modifications which we discuss in this section.
Baselines We started by designing three baseline architectures. The first baseline produced predictions solely
from the question while totally ignoring the image. The

78.56
78.64
79.90

27.98
32.98
36.96

30.76
39.79
49.58

All
48.3
54.82
59.34

Table 1. Evaluation of the baseline models on the VQA 1.0 validation split.

model used the same question representation described in
[7] and passed the output to a softmax 3000-way classification layer. The goal of this architecture was to assess the
extent of the language bias present in VQA.
The second baseline is a simple joint representation of
the image features and the language representation. The
representations were combined using the compact bilinear
pooling from [25]. We chose this method specifically because it was shown to be effective by Fukui et al. [7]. The
main objective of this model is to measure how a robust
pooling method of multimodal features would perform on
its own without a deep architecture or attention. We refer to
this model as Simple MCB.
For the last baseline, we substituted the compact bilinear pooling from Simple MCB with an LSTM consisting
of hidden states equal to the image size. A 1 × 1 convolutional layer followed by a tanh activation were used on the
image features prior to the LSTM, while the question representation was replicated to have a common embedding size
for both representations This model is referred to as Joint
LSTM.
We begin by testing our baseline models on the VQA
1.0 validation set. As shown in Table 1, the language-only
baseline model managed to get 48.3% overall. More impressively, it scored 78.56% on Yes/No questions. The Simple MCB model further improves the overall performance,
although little improvement is gained in the binary Yes/No
tasks. Replacing MCB with our basic Joint LSTM embedding improves performance across the board.
Modifications to the Joint LSTM Model We test several variations of the Joint LSTM baseline which are highlighted in Table 2. Using PReLU activations has helped
in two ways. First, it reduced time for convergence from
240K iterations to 120K. Second, the overall accuracy has
improved, especially in the Other category. The next modifications were inspired by the results from [18]. We experimented with appending positional features which can be described as the coordinates of each pixel to the depth/feature
dimension of the image representation. When unnormalized
with respect to the other features, it worsened results significantly, dropping the overall accuracy by over 2 points.

Model

VQA 1.0 Validation Split
Open Ended Task
Y/N Num. Other

Joint LSTM baseline
PReLU
Pos. features
Pos. features (norm.)
High dropout
Extra FC

79.90
79.61
79.68
79.69
79.03
78.86

36.96
36.21
36.52
36.36
34.84
33.51

49.58
50.77
46.59
50.69
47.25
45.57

All
59.34
59.74
57.71
59.75
57.59
56.51

Table 2. Evaluation of the Joint LSTM model and its modifications
on the VQA 1.0 validation split.

Normalizing positional features did not have enough of a
noticeable improvement (0.01 points overall) to warrant its
effectiveness. Next, all dropout values are increased from
0.3 to 0.5 deteriorated the network’s accuracy, particularly
in the Number and Other categories. The final modification was inserting a fully connected layer with 1024 hidden
units before the classifier, which surprisingly dropped the
accuracy massively.

4.2. VQA 2.0 Experiments
After the release of VQA 2.0, we shifted our empirical
evaluation towards the newer dataset. First, we retrain and
retest our best performing VQA 1.0 model Joint LSTM as
well as several improvements and modifications.
Since VQA 2.0 was built to reduce the language prior
and bias inherent in VQA, the accuracy of Joint LSTM drops
significantly as shown in Table 3. Note that all the models that were trained so far do not have explicit visual or
textual attention implemented. Our first network with explicit visual attention, RVAU, shows an accuracy jump by
almost 3 points compared to the Joint LSTM model. This
result highlights the importance of attention for good performance in VQA. Training the RVAU network as a multilabel task (RVAUmultilabel ), i.e. using all available annotations
at each training iteration, drops the accuracy horribly. This
is the biggest drop in performance so far. This might be
caused by the variety of annotations in VQA for each question which makes the task for optimizing all answers at once
much harder.
DRAU Evaluation The addition of RTAU marks the creation of our DRAU network. The DRAU model shows favorable improvements over the RVAU model. Adding textual attention improves overall accuracy by 0.56 points.
Substituting the PReLU activations with ReLU (DRAUReLU )
massively drops performance. While further training might
have helped the model improve, PReLU offers much faster
1 Concurrent

Work

Model

VQA 2.0 Validation Split
Open Ended Task
Y/N Num.

Other

All

Joint LSTM w/PReLU

72.04

37.95

48.58

56.00

RVAU
RVAUmultilabel

74.59
77.53

37.75
36.05

52.81
40.18

59.02
53.67

DRAUHadamard fusion
DRAUanswer vocab = 5k
DRAUReLU
DRAUno final dropout
DRAUhigh final dropout

76.62
76.33
72.69
77.02
76.47

38.92
38.21
34.92
38.26
38.71

52.09
51.85
45.05
50.17
52.52

59.58
59.27
54.11
58.69
59.71

-

-

-

59.14
59.67

MCB [26]
Kazemi and Elqursh [18]1

Table 3. Evaluation of RVAU and DRAU-based models on the
VQA 2.0 validation split.

convergence. Increasing the value of the dropout layer after
the fusion operation (DRAUhigh final dropout ) improves performance by 0.13 points, in contrast to the results of the Joint
LSTM model on VQA 1.0. Note that on the VQA 1.0 tests,
we changed the values of all layers that we apply dropout
on, but here we only change the last one after the fusion
operation. Totally removing this dropout layer worsens accuracy. This suggests that the optimal dropout value should
be tuned per-layer.
We test a few variations of DRAU on the test-dev set. We
can observe that VQA benefits from more training data; the
same DRAU network performs better (62.24% vs. 59.58%)
thanks to the additional data. Most of the literature resize
the original ResNet features from 224 × 224 to 448 × 448.
To test the effect of this scaling, we train a DRAU variant
with the original ResNet size (DRAUsmall ). Reducing the
image feature size from 2048 × 14 × 14 to 2048 × 7 × 7
adversely affects accuracy as shown in Table 4. Adding
more glimpses significantly reduces the model’s accuracy
(DRAUglimpses = 4 ). A cause of this performance drop could
be related to the fact that LSTMs process the input in a onedimensional fashion and thus decide that each input is either
relevant or non-relevant. This might explain why the attention maps of DRAU separate the objects from the background in two glimpses as we will mention in Section 5.
2D Grid LSTMs [31] might help remove this limitation.
Removing the extra data from Visual Genome hurts the
model’s accuracy. That supports the fact that VQA is very
diverse and that extra data helps the model perform better.
Finally, substituting Hadamard product of MCB in the final
fusion operation boosts the network’s accuracy significantly
by 1.17 points (DRAUMCB fusion ).
As mentioned in Section 3.1, we experiment replacing
the global ResNet features with object-level features as sug-

VQA 2.0 Test-Dev Split
Open Ended Task
Y/N
Num. Other

Model

DRAUHadamard fusion
DRAUsmall
DRAUglimpses = 4
DRAUno genome
DRAUMCB fusion
DRAUFRCNN features

78.27
77.53
76.82
79.63
78.97
82.85

40.31
38.78
39.15
39.55
40.06
44.78

53.57
49.93
51.07
51.81
55.47
57.4

4.4. DRAU versus the state-of-the-art
All
62.24
60.03
60.32
61.88
63.41
66.45

Table 4. Evaluation of later DRAU-based models on the VQA 2.0
test-dev split.

VQA 2.0 Test-dev Split
Open Ended Task
Y/N Num. Other

Model

All

3

MCB [7]
MCB w/RVAU

78.41
77.31

38.81
40.12

53.23
54.64

61.96
62.33

MUTAN [14]
MUTAN w/RVAU

79.06
79.33

38.95
39.48

53.46
53.28

62.36
62.45

Table 5. Results of state-of-the-art models with RVAU.

gested by [22]. This change provides a significant performance increase of 3.04 points (DRAUFRCNN features ).

4.3. Transplanting RVAU in other models
To verify the effectiveness of the recurrent attention
units, we replace the attention layers in MCB and MUTAN
[14] with RVAU.
For MCB [7] we remove all the layers after the first MCB
operation until the first 2048-d output and replace them with
RVAU. Due to GPU memory constraints, we reduced the
size of each hidden unit in RVAU’s LSTM from 2048 to
1024. In the same setting, RVAU significantly helps improve the original MCB model’s accuracy as shown in Table 5. The most noticeable performance boost can be seen
in the number category, which supports our hypothesis that
recurrent layers are more suited for sequential reasoning.
Furthermore, we test RVAU in the MUTAN model [14].
The authors use a multimodal vector with dimension size of
510 for the joint representations. For coherence, we change
the usual dimension size in RVAU to 510. At the time of this
writing, the authors have not released results on VQA 2.0
using a single model rather than a model ensemble. Therefore, we train a single-model MUTAN using the authors’
implementation.2 The story does not change here, RVAU
improves the model’s overall accuracy.
2 https://github.com/Cadene/vqa.pytorch
3 http://www.visualqa.org/roe_2017.html

VQA 1.0 Table 6 shows a comparison between DRAU
and other state-of-the-art models. Excluding model ensembles, DRAU performs favorably against other models. To
the best of our knowledge, [5] has the best single model
performance of 65.07% on the test-std split which is very
close our best model (65.03%). Small modifications or hyperparameter tuning could push our model further. Finally,
the FRCNN image features boosts the model’s performance
close to the state-of-the-art ensemble model.
VQA 2.0 Our model DRAUMCB fusion landed the 8th place
in the VQA 2.0 Test-standard task.4 . Currently, all reported
submissions that outperform our single model use model
ensembles. Using FRCNN features boosted the model’s
performance to outperform some of the ensemble models
(66.85%). The first place submission [22] reports using
an ensemble of 30 models. In their report, the best single
model that uses FRCNN features achieves 65.67% on the
test-standard split which is outperformed by our best single
model DRAUFRCNN features .

5. DRAU versus MCB
In this section, we provide qualitative results that highlight the effect of the recurrent layers compared to the MCB
model.
The strength of RAUs is notable in tasks that require sequentially processing the image or relational/multi-step reasoning. In the same setting, DRAU outperforms MCB in
counting questions. This is validated in a subset of the validation split questions in the VQA 2.0 dataset as shown in
Figure 4. Figure 5 shows some qualitative results between
DRAU and MCB. For fair comparison we compare the first
attention map of MCB with the second attention map of our
model. We do so because the authors of MCB [7] visualize
the first map in their work5 . Furthermore, the first glimpse
of our model seems to be the complement of the second
attention, i.e. the model separates the background and the
target object(s) into separate attention maps. We have not
tested the visual effect of more than two glimpses on our
model.
In Figure 5, it is clear that the recurrence helps the model
attend to multiple targets as apparent in the difference of the
attention maps between the two models. DRAU seems to
also know how to count the right object(s). The top right example in Figure 5 illustrates that DRAU is not easily fooled
by counting whatever object is present in the image but
rather the object that is needed to answer the question. This
4 https://evalai.cloudcv.org/web/challenges/
challenge-page/1/leaderboard
5 https://github.com/akirafukui/vqa-mcb/blob/
master/server/server.py#L185

Model

VQA 1.0 Open Ended Task
Test-dev
Y/N Num. Other
All
Y/N

Test-standard
Num. Other

SAN [10]
DMN+ [16]
MRN [21]
HieCoAtt [9]
RAU [8]
DAN [6]
MCB [7] (e = 7)
MLB [5] (1 model)
MLB [5] (e = 7)
MUTAN [14] (e = 5)

79.3
80.5
82.28
79.7
81.9
83.0
83.4
84.57
85.14

36.6
36.8
38.82
38.7
39.0
39.1
39.8
39.21
39.81

46.1
48.3
49.25
51.7
53.0
53.9
58.5
57.81
58.52

58.7
60.3
61.68
61.8
63.3
64.3
66.7
66.77
67.42

82.39
81.7
82.8
83.24
84.02
84.61
84.91

38.23
38.2
38.1
39.47
37.90
39.07
39.79

49.41
52.8
54.0
58.00
54.77
57.79
58.35

58.9
60.4
61.84
62.1
63.2
64.2
66.47
65.07
66.89
67.36

DRAUHadamard fusion
DRAUMCB fusion
DRAUFRCNN features

82.73
82.44
84.92

38.18
38.22
39.16

54.43
56.30
57.70

64.3
65.1
66.86

82.41
84.87

38.33
40.02

55.97
57.91

65.03
67.16

All

Table 6. DRAU compared to the state-of-the-art on the VQA 1.0 dataset.e = n corresponds to a model ensemble of size n.

Model

VQA 2.0 Open Ended Task
Test-dev
Y/N Num. Other
All

Y/N

Test-standard
Num. Other

UPC
MIC TJ
neural-vqa-attention[10]
CRCV REU
VQATeam MCB [26]
DCD ZJU[32]
VQAMachine [33]
POSTECH
UPMC-LIP6[14]
LV NUS[34]
DLAIT
HDU-USYD-UNCC[35]
Adelaide-Teney ACRV MSR[36]

67.1
69.02
70.1
73.91
78.41
79.84
79.4
78.98
81.96
81.95
82.94
84.39
85.24

31.54
34.52
35.39
36.82
38.81
38.72
40.95
40.9
41.62
48.31
47.08
45.76
48.19

25.46
35.76
47.32
54.85
53.23
53.08
53.24
55.35
57.07
59.99
59.94
59.14
59.88

43.3
49.32
55.35
60.65
61.96
62.47
62.62
63.45
65.57
67.71
67.95
68.02
69.00

66.97
69.22
69.77
74.08
78.82
79.85
79.82
79.32
82.07
81.92
83.17
84.5
85.54

31.38
34.16
35.65
36.43
38.28
38.64
40.91
40.67
41.06
48.38
46.66
45.39
47.45

25.81
35.97
47.18
54.84
53.36
52.95
53.35
55.3
57.12
59.63
60.15
59.01
59.82

43.48
49.56
55.28
60.81
62.27
62.54
62.97
63.66
65.71
67.64
68.22
68.09
69.13

DRAUHadamard fusion
DRAUMCB fusion
DRAUFRCNN features

78.27
78.97
82.85

40.31
40.06
44.78

53.58
55.48
57.4

62.24
63.41
66.45

78.86
79.27
83.35

39.91
40.15
44.37

53.76
55.55
57.63

62.66
63.71
66.85

All

Table 7. DRAU compared to the current submissions on the VQA 2.0 dataset.

property also translates to questions that require relational
reasoning. The second column in Figure 5 demonstrates
how DRAU can attend the location required to answer the
question based on the textual and visual attention maps.

6. Conclusion
We proposed an architecture for VQA with a novel attention unit, termed the Recurrent Attention Unit (RAU).
The recurrent layers help guide the textual and visual atten-

tion since the network can reason relations between several
parts of the image and question. We provided quantitative
and qualitative results indicating the usefulness of a recurrent attention mechanism. Our DRAU model showed improved performance in tasks requiring sequential/complex
reasoning such as counting or relational reasoning over [7],
the winners of the VQA 2016 challenge. In VQA 1.0, we
achieved near state-of-the-art results for single model performance with our DRAU network (65.03 vs. 65.07 [5]).
Adding the FRCNN features gets the model within margin

Question Type (Occurrence)

Accuracy(%)

47.28

46.31
43.98

42.06

DRAU
MCB
44.30
42.57

Visual Question Answering and Visual Grounding,” in Empirical Methods in Natural Language Processing (EMNLP),
2016, pp. 457–468.
[8] H. Noh and B. Han, “Training Recurrent Answering Units
with Joint Loss Minimization for VQA,” arXiv:1606.03647,
2016.
[9] J. Lu, J. Yang, D. Batra, and D. Parikh, “Hierarchical
Question-Image Co-Attention for Visual Question Answering,” in Advances in Neural Information Processing Systems
(NIPS), 2016, pp. 289–297.

how many
people are in
(905)

how many
people are
(2,005)

how many
(20,462)

Figure 4. Results on questions that require counting in the VQA
2.0 validation set.

of the state-of-the-art 5-model ensemble MUTAN [14]. Finally, we demonstrated that substituting the visual attention
mechanism in other networks, MCB [7] and MUTAN [14],
consistently improves their performance. In future work we
will investigate implicit recurrent attention mechanism using recently proposed explanation methods [37, 38].

References
[1] K. He, X. Zhang, S. Ren, and J. Sun, “Delving Deep into
Rectifiers: Surpassing Human-Level Performance on ImageNet Classification,” in IEEE International Conference on
Computer Vision (ICCV), 2015, pp. 1026–1034.
[2] S. Bosse, D. Maniry, K.-R. Müller, T. Wiegand, and
W. Samek, “Deep neural networks for no-reference and fullreference image quality assessment,” IEEE Transactions on
Image Processing, vol. 27, no. 1, pp. 206–219, 2018.

[10] Z. Yang, X. He, J. Gao, L. Deng, and A. Smola, “Stacked Attention Networks for Image Question Answering,” in IEEE
Conference on Computer Vision and Pattern Recognition
(CVPR), 2016, pp. 21–29.
[11] K. Chen, J. Wang, L.-C. Chen, H. Gao, W. Xu, and
R. Nevatia, “ABC-CNN: An Attention Based Convolutional Neural Network for Visual Question Answering,”
arXiv:1511.05960, 2015.
[12] H. Xu and K. Saenko, “Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question
Answering,” in European Conference on Computer Vision
(ECCV), 2016, pp. 451–466.
[13] M. Charikar, K. Chen, and M. Farach-Colton, “Finding frequent items in data streams,” Theoretical Computer Science,
vol. 312, no. 1, pp. 3–15, 2004.
[14] H. Ben-younes, R. Cadene, M. Cord, and N. Thome, “MUTAN: Multimodal Tucker Fusion for Visual Question Answering,” arXiv:1705.06676, 2017.
[15] L. R. Tucker, “Some mathematical notes on three-mode factor analysis,” Psychometrika, vol. 31, no. 3, pp. 279–311,
1966.

[3] D. Bahdanau, K. Cho, and Y. Bengio, “Neural Machine
Translation by Jointly Learning to Align and Translate,”
in International Conference on Representation Learning
(ICLR), 2015.

[16] C. Xiong, S. Merity, and R. Socher, “Dynamic Memory Networks for Visual and Textual Question Answering,” in International Conference on Machine Learning (ICML), 2016,
pp. 2397–2406.

[4] R. Nallapati, B. Zhou, C. Gulcehre, B. Xiang et al., “Abstractive text summarization using sequence-to-sequence rnns
and beyond,” arXiv:1602.06023, 2016.

[17] P. R. Kumar and P. Varaiya, Stochastic systems: Estimation,
identification, and adaptive control. SIAM, 2015.

[5] J.-H. Kim, K.-W. On, J. Kim, J.-W. Ha, and B.-T. Zhang,
“Hadamard Product for Low-rank Bilinear Pooling,” in International Conference on Representation Learning (ICLR),
2017.
[6] H. Nam, J.-W. Ha, and J. Kim, “Dual Attention Networks
for Multimodal Reasoning and Matching,” in IEEE International Conference on Computer Vision (CVPR), 2017, pp.
299–307.
[7] A. Fukui, D. H. Park, D. Yang, A. Rohrbach, T. Darrell, and
M. Rohrbach, “Multimodal Compact Bilinear Pooling for

[18] V. Kazemi and A. Elqursh, “Show, Ask, Attend, and Answer: A Strong Baseline For Visual Question Answering,”
arXiv:1704.03162, 2017.
[19] Q. Wu, P. Wang, C. Shen, A. Dick, and A. van den Hengel, “Ask Me Anything: Free-form Visual Question Answering Based on Knowledge from External Sources,” in IEEE
Conference on Computer Vision and Pattern Recognition
(CVPR), 2016, pp. 4622–4630.
[20] A. Ray, G. Christie, M. Bansal, D. Batra, and D. Parikh,
“Question Relevance in VQA: Identifying Non-Visual And
False-Premise Questions,” in Empirical Methods in Natural
Language Processing (EMNLP), 2016, pp. 919–924.

How many

How many
camels are in

lanterns hang

the photo?
DRAU: 0

off the clock
tower ?
DRAU: 4
How many
lanterns hang
off the clock
tower?
GT: 4

MCB: 1

How many
camels are in
the photo?
GT: 0

What is on the
floor leaning on
the bench in

How many
people are

between the

shown ?
DRAU: 3

How many
people are
shown?
GT: 3

MCB: 2

people ?
DRAU: racket
What is on the
floor leaning on
the bench in
between the
people?
GT: racket

does the vase

animals are

depict ?
DRAU: owl

there ?
DRAU: 2

MCB: one

MCB: backpack

What animal

How many

How many
animals are
there?
GT: 2

MCB: 1

What animal
does the vase
depict?
GT: owl

MCB: elephant

Figure 5. DRAU vs. MCB Qualitative examples. Attention maps for both models shown, only DRAU has textual attention.

[21] J.-H. Kim, S.-W. Lee, D. Kwak, M.-O. Heo, J. Kim, J.-W.
Ha, and B.-T. Zhang, “Multimodal Residual Learning for
Visual QA,” in Advances in Neural Information Processing
Systems 29, 2016, pp. 361–369.
[22] P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson,
S. Gould, and L. Zhang, “Bottom-Up and Top-Down At-

tention for Image Captioning and VQA,” arXiv:1707.07998,
2017.

[23] S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN:
Towards Real-Time Object Detection with Region Proposal
Networks,” arXiv:1506.01497, Jun. 2015.

[24] J. Pennington, R. Socher, and C. D. Manning, “GloVe:
Global Vectors for Word Representation,” in Empirical
Methods in Natural Language Processing (EMNLP), 2014,
pp. 1532–1543.
[25] Y. Gao, O. Beijbom, N. Zhang, and T. Darrell, “Compact
bilinear pooling,” in IEEE Conference on Computer Vision
and Pattern Recognition (CVPR), 2016, pp. 317–326.
[26] Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and
D. Parikh, “Making the V in VQA Matter: Elevating the
Role of Image Understanding in Visual Question Answering,” in IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), 2017, pp. 6904–6913.
[27] S. Antol, A. Agrawal, J. Lu, M. Mitchell, C. L. Zitnick,
D. Batra, and D. Parikh, “VQA: Visual Question Answering,” in IEEE International Conference on Computer Vision
(CVPR), 2015, pp. 2425–2433.
[28] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in European Conference on Computer Vision – ECCV 2014, 2014, pp. 740–755.
[29] D. P. Kingma and J. Ba, “Adam: A Method for Stochastic
Optimization,” arXiv:1412.6980, 2014.
[30] X. Glorot and Y. Bengio, “Understanding the difficulty of
training deep feedforward neural networks,” in AISTATS,
2010, pp. 249–256.
[31] N. Kalchbrenner, I. Danihelka, and A. Graves, “Grid long
short-term memory,” arXiv:1507.01526, 2015.
[32] Y. Lin, Z. Pang, D. Wang, and Y. Zhuang, “Task-driven Visual Saliency and Attention-based Visual Question Answering,” arXiv:1702.06700, 2017.
[33] P. Wang, Q. Wu, C. Shen, and A. van den Hengel, “The
VQA-Machine: Learning How to Use Existing Vision Algorithms to Answer New Questions,” arXiv:1612.05386, 2016.
[34] I. Ilievski and J. Feng, “A Simple Loss Function for Improving the Convergence and Accuracy of Visual Question Answering Models,” arXiv:1708.00584, 2017.
[35] Z. Yu, J. Yu, C. Xiang, J. Fan, and D. Tao, “Beyond Bilinear:
Generalized Multi-modal Factorized High-order Pooling for
Visual Question Answering,” arXiv:1708.03619, 2017.
[36] D. Teney, P. Anderson, X. He, and A. van den Hengel, “Tips
and Tricks for Visual Question Answering: Learnings from
the 2017 Challenge,” arXiv:1708.02711, 2017.
[37] G. Montavon, W. Samek, and K.-R. Müller, “Methods for interpreting and understanding deep neural networks,” Digital
Signal Processing, vol. 73, pp. 1–15, 2018.
[38] L. Arras, G. Montavon, K.-R. Müller, and W. Samek, “Explaining recurrent neural network predictions in sentiment
analysis,” in EMNLP’17 Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis
(WASSA), 2017, pp. 159–168.

