Visual Question Generation as Dual Task of Visual Question Answering
Yikang Li1 , Nan Duan2 , Bolei Zhou3 , Xiao Chu1 , Wanli Ouyang4 , Xiaogang Wang1

arXiv:1709.07192v1 [cs.CV] 21 Sep 2017

1

The Chinese University of Hong Kong, Hong Kong, China 2 Microsoft Research Asia, China
3
Massachusetts Institute of Technology, USA 4 University of Sydney, Australia

Abstract
Recently visual question answering (VQA) and visual question generation (VQG) are two trending topics in the computer vision, which have been explored separately. In this
work, we propose an end-to-end unified framework, the Invertible Question Answering Network (iQAN), to leverage
the complementary relations between questions and answers
in images by jointly training the model on VQA and VQG
tasks. Corresponding parameter sharing scheme and regular terms are proposed as constraints to explicitly leverage
hQ,Ai’s dependencies to guide the training process. After
training, iQAN can take either question or answer as input,
then output the counterpart. Evaluated on the large scale visual question answering datasets CLEVR and VQA2, our
iQAN improves the VQA accuracy over the baselines. We
also show the dual learning framework of iQAN can be generalized to other VQA architectures and consistently improve
the results over both the VQA and VQG tasks.1

encoder
question

𝒒

image

𝒗

answer

𝒂

image

𝒗

fusion

decoder

ෝ
𝒂

answer

ෝ
𝒒

question

Figure 1: Problem solving schemes of VQA (top) and
VQG (bottom), both of which utilize the hencoder-fusiondecoderi pipeline with Q and A in inverse order. v, q and
a respectively denote the encoded features of input image,
question, and answer, while â and q̂ represent the predicted
answer/question features.

Introduction
Question answering (QA) and question generation (QG)
are two fundamental tasks in natural language processing (Manning, Schütze, and others 1999; Martin and Jurafsky 2000). In recent years, computer vision has been
introduced so that they become cross-modality tasks, Visual Question Answering (VQA) (Zhou et al. 2015; Yu et
al. 2017; Antol et al. 2015) and Visual Question Generation (VQG) (Mostafazadeh et al. 2016; Zhang et al. 2016),
which refer to the techniques of the computer vision and the
natural language processing. Both VQA and VQG tasks involve reasoning between a question text and an answer text
based on the content of the given image. The task of VQA
is to answer image-based questions, while the VQG aims at
generating reasonable questions based on the image content
and the given answer.
In previous works, VQA and VQG are studied separately.
As shown in Figure 1, the VQA model encodes the question sentence as an embedding q, then associates q with the
image feature v to infer the answer embedding â, which is
decoded as the distribution over the answer vocabulary. Different from VQA, VQG has not got a standard problem setCopyright c 2018, Association for the Advancement of Artificial
Intelligence (www.aaai.org). All rights reserved.
1
Source code will be released when accepted.

ting. In this work, we formulate VQG as generating a question with an image and answer given. where the VQG model
merges the answer embedding a and the image feature v to
get the question embedding q̂, and then generates a question sentence with recurrent neural network (RNN). We can
see that these two tasks are intrinsically correlated, i.e. sharing visual input and taking encoder-fusion-decoder pipeline
with Q and A in reverse order. Thus, we refer them as “Dual”
tasks.
Duality reflects the intrinsic complementary relation between question answering and generation. Intuitively, learning to answer questions may boost the question generation
and vice versa, as both of them require similar abilities: image recognition, question reasoning, cross-modal information association, etc. Thus, we argue that jointly learning
through these two tasks can utilize the training data in a more
efficient way, and can bring mutual improvements to both
VQA and VQG. Therefore, we formulate the dual training
of VQA and VQG as learning an invertible cross-modality
fusion model that can infer Q or A when given the counterpart based on the given image.
From this perspective, we derive an invertible Dual Mutan fusion module, based on the state-of-the-art VQA model

Mutan (Ben-younes et al. 2017). The module can complete
the feature inference in a bidirectional manner, i.e. it can infer the answer embeddings from image+question and infer
the question embeddings from image+answer. Furthermore,
by sharing the visual features as well as the encoder and decoder of the question and answer, VQG and VQA models
can be viewed as the inverse form of each other with parameters shared. When jointly training on the two tasks, the
invertibility brought by our parameter sharing schemes can
help to regularize the training process, and multiple training
tasks will help the model learn more general representations.
Contribution: This work is the first attempt to consider
VQG and VQA as dual tasks and formulate them into a unified framework called Invertible Question Answering Network (iQAN). The model is jointly trained with VQA and
VQG tasks and can be deployed for either task in the testing stage. In the iQAN, a novel parameter sharing scheme
and duality regularization are proposed to explicitly leverage the complementary relations between questions and answers. Evaluated on VQA2 and CLEVR datasets, our proposed model achieves better results on VQA task. Experimental results show that our framework can also generalize to some other VQA models and continuously improves
their performance. Besides, we propose a method to utilize
the VQG model to augment questions with ground-truth answers given, which could employ the cheaply-labeled answers to boost the model training.

Related Work
Visual Question Answering (VQA) is one of the most
popular cross-discipline tasks aiming at understanding the
question and image, and then providing the correct answer.
Malinowski et al. propose an encoder-decoder framework
to merge the visual and textual information for answering
prediction (Malinowski, Rohrbach, and Fritz 2015). Shih et
al. introduce visual attention mechanism to highlight the
image regions relevant to answering the question (Shih,
Singh, and Hoiem 2016). Lu et al. further apply attention
to the language model, called co-attention, to jointly reason about images and questions (Lu et al. 2016). Apart from
proposing new frameworks, some focus on designing effective multimodal feature fusion schemes (Fukui et al. 2016;
Kim et al. 2017). The bilinear model Mutan proposed by
Ben-younes et al. is the state-of-the-art method to model interactions between two modalities (Ben-younes et al. 2017).
Additionally, several benchmark dataset are proposed to facilitate the VQA research (Malinowski and Fritz 2014).
VQA2 is the most popular open-ended Q-A dataset (Goyal
et al. 2017) with real images. Johnson et al. propose CLEVR
dataset with rendered images and automatically-generated
questions to mitigate answer biases and diagnose the reasoning ability of VQA models. We will evaluate our method
on these two datasets.
Visual Question Generation. Question generation from
text corpus has been investigated for years in natural language processing (Ali, Chali, and Hasan 2010; Kalady,
Elikkottil, and Das 2010; Serban et al. 2016). Recently,
it has been introduced to computer vision to generate
image-related questions. Mora et al. propose a CNN-LSTM

model to directly generate image-related questions and
corresponding answers (Mora, de la Puente, and Giro-i
Nieto 2016). Mostafazadeh et al. collect the first VQG
dataset, where each image is annotated with several questions (Mostafazadeh et al. 2016). Zhang et al. propose a
model to automatically generate visually grounded questions (Zhang et al. 2016), which uses Densecap (Johnson,
Karpathy, and Fei-Fei 2015) to generate region captions as
extra information to guide the question generation. Jain et al.
combine the variational autoencoder and LSTM to generate
diverse questions (Jain, Zhang, and Schwing 2017). Different from the existing works to generate question solely based
on images, we provide an annotated answer as an additional
cue. Therefore, VQG can be modeled as a two modalities
fusion problem like VQA.
Dual Learning. Utilizing cycle consistency to regularize
the training process has a long history. It has been use as a
standard trick for years in visual tracking to enforce forwardbackward consistency (Sundaram, Brox, and Keutzer 2010).
He et al. first formulate the idea as Dual Learning in machine translation area (He et al. 2016), which uses A-to-B
and B-to-A translation models to form a closed translation
loop (A-B-A and B-A-B), and lets them teach each other
through a reinforcement learning process. Tang et al. introduce the idea to QA area, where question generation is
modeled as dual task of QA, and leverage the probabilistic
correlation between QA and QG to guide the training (Tang
et al. 2017). Zhu et al. employ the idea in computer vision
and propose the image translation model CycleGAN (Zhu et
al. 2017). However, till now there is no existing work that
utilizes dual learning on VQA. Hence, our work is the first
attempt to model the VQA and VQG as dual tasks and leverage the complementary relations between the two tasks.

Invertible Question Answering Network
(iQAN)
In this section, we present the dual learning framework of
the VQA and VQG, Invertible Question Answering Network (iQAN). The overview of our proposed iQAN is shown
in Figure 2, which consists of two components, VQA component (top) and VQG component (bottom).
In the VQA component, given a question q, LSTM is
used for obtaining the embedded features q ∈ Rdq , and
CNN is used transformed the input image v into an feature
map. A Mutan-based attention module is used to generate
a question-related visual feature vq ∈ Rdv from the image
and the question. Then a Mutan fusion module is used for
obtaining the answer features â ∈ Rda from vq and q. Finally, a linear classifier Wa is used to predict the answer for
VQA.
In the VQG component, given an answer, a lookup table
Ea is used for obtaining the embedded feature a ∈ Rda .
CNN with attention module is used for obtaining the visual feature va ∈ Rdv from the input image and the answer
feature a. Then the Mutan in the dual form, which shares
parameters with VQA Mutan but in a different structure, is
used for obtaining the predicted question features q̂ ∈ Rdq .
Finally, an LSTM-based decoder is employed to translate q̂

𝒒
What color is the
batter’s jersey

ෝ
𝒂
A duality

Mutan

LSTM

𝑾𝑎
Attention
VQA loss

𝒗𝑞
ResNet 152

Green
What color is the
batter’s jersey

Attention

Green

𝑬𝒂 = 𝑾⊤
𝑎

𝒂

Loss

ෝ
𝒒

VQG loss

LSTM

𝒗𝑎

Mutan∗

Q duality

Figure 2: Overview of Invertible Question Answering Network (iQAN), which consists two parts, VQA and VQG. The upper
part is Mutan VQA component (Ben-younes et al. 2017), and the lower part is its dual VQG model. Input questions and answers
are encoded respectively by an LSTM and a lookup table Ea into fixed-length features. With attention and Mutan fusion
module, predicted features are obtained. The predict features are used for obtaining output (by LSTM and Wa for questions
and answers respectively). A duality and Q duality are duality regularizers to constrain the similarity between the answer and
question representations in both models. Two components share the LSTM, Mutan and Attention Modules. Mutan∗ denotes
the dual form of Mutan. Ea also shares parameters with Wa .
to the question sentence.
We will formulate the VQA and VQG components as inverse process to each other by introducing a novel parameter sharing scheme and the duality regularizer. Therefore,
we could jointly train one model with two tasks to leverage
the dependencies of questions and answer in a bidirectional
way. In addition, the invertibility of the model could serve
as a regular term to guide the training process.

The VQA component

denotes the mode-i product between a tensor and a matrix:
(T ×i U) [d1 , ...di−1 , j, di+1 ...dN ] =

Di
X

T [d1 ...dN ]U[di , j]

di =1

(2)
To reduce the complexity of the full tensor T , Tucker decomposition (Ben-younes et al. 2017) is introduced as an effective way to factorize T as a tensor product between factor
matrices Wq , Wv and Wa , and a core tensor Tc :
T = ((Tc ×1 Wq ) ×2 Wv ) ×3 Wa

The VQA component of our proposed iQAN is based on the
state-of-the-art Mutan VQA model. We will briefly review
the core part, Mutan fusion module, which takes an image
feature vq and a question feature q as input, and predicts the
answer feature â.

tq ×dq

tv ×dv

(3)
ta ×da

with Wq ∈ R
, Wv ∈ R
and Wa ∈ R
, and
Tc ∈ Rtq ×tv ×ta . Consequently, we can rewrite Eq. 1 as:
â = ((Tc ×1 (Wq q)) ×2 (Wv vq )) ×3 Wa

(4)

Review on MUTAN fusion module Since language and
visual representations are in different modalities, the issue
of merging visual and linguistic features is crucial in VQA.
Bilinear models are recent powerful solutions to the multimodal fusion problem, which encode bilinear interactions
between q and vq as follows:

where matrices Wq and Wv transform the question features
q and image features vq into dimensions tq and tv respectively. The squeezed bilinear core Tc models the interactions
among the transformed features and projects them to the answer space of size ta , which is used to infer the per-class
score by Wa .
If we define q̃ = Wq q ∈ Rtq and ṽq = Wv vq ∈ Rtv ,
then we have:

â = (T ×1 q) ×2 vq

ã = (Tc ×1 q̃) ×2 ṽq ∈ Rta

dq ×dv ×da

(1)

where the tensor T ∈ R
denotes the fullyparametrized operator for answer feature inference, and ×i

(5)

Thus, ã can be viewed as the answer feature where â =
ã> × Wa .

𝒒

To balance the complexity and expressivity of the interaction modeling, the low rank assumption is introduced, and
Tc [:, :, k] can be expressed as a sum of R rank one matrices:
Tc [:, :, k] =

R
X

𝑾𝒒

>

𝒂

with mkr ∈ Rtq , nkr ∈ Rtv and ⊗ denoting the outer product. Then each element ã [k] of ã, k ∈ {1, . . . , ta }, can be
written as:
R
X

q̃> mkr



ṽq> nkr

ෝ
𝒂

(6)

r=1

ã [k] =

𝑾𝒂

𝑾𝒗

𝒗

mkr ⊗ nkr

Primal Form



Dual Form

𝑾𝒂
𝒗

𝑾𝒗

𝑾𝒒

ෝ
𝒒

(7)

r=1

We can define R matrices Mr ∈ Rtq ×ta and Nr ∈ Rtv ×ta
such that Mr [:, k] = mkr and Nr [:, k] = nkr . Therefore,
with sparsity constraints, Eq. 5 is further simplified as:
ã =

R
X

q̃> Mr



ṽq> Nr



(8)

r=1

Dual MUTAN

where
denotes the element-wise product. With Mutan,
low computational complexity and strong expressivity of the
model are both obtained.

The VQG component
The VQG component of our proposed iQAN is formulated
as generating a question (word sequence) given an image
and an answer label.
During training, our target is to learn a model such that
the generated question q̂ similar to the referenced one q ∗ .
The generation of each word of the question can be written
as:

(9)
ŵt = arg max p w v, w0 , ..., wt−1
w∈W

where W denotes the word vocabulary. ŵt is the predicted
word at t step. wi represents the i-th ground-truth word.
Beam search will be used during inference.
VQG shares the visual CNN with VQA part. The answer
feature a ∈ Rda is directly retrieved from the answer embedding table Ea . MUTAN is also utilized for visual attention
module and visual & answer representations fusion at VQG.
Similar to Eq. 8, the inference of question features q̃ can be
formulated as:
q̃ =

R
X

Figure 3: The Dual Mutan in the primal form (for VQA) and
dual form (for VQG). The two forms share one parameter
set: the core tensor T c , projection matrices of images, questions, and answers. In our experiment, Wa at the top part
and Wq at the bottom part are merged with decoders.

ã> M0r



ṽa> N0r



(10)

r=1

with ã = Wa a ∈ Rta and ṽa = Wv va ∈ Rtv . M0r and N0r
are defined as Eq. 8. Finally, the predicted question features
q̃ is fed into an RNN-based model to generate the predicted
question.
From the formulation in (8) and (10), the VQG Mutan
could be viewed as the conjugate form of the VQA Mutan. In
the next section, we will introduce our attempt to investigate
the connection between the two Mutan modules.

To leverage the duality of questions and answers, we derive a Dual Mutan from the original Mutan to finish the primal (question-to-answer) and its dual (answer-to-question)
inference on the feature level with one kernel.
First we rewrite Eq. 5 and its dual form:
ã∗ = (Tc ×1 q̃) ×2 ṽ
q̃∗ = (Tc0 ×1 ã) ×2 ṽ0

(11)

where Tc ∈ Rtq ×tv ×ta , Tc0 ∈ Rta ×tv ×tq , q̃ = Wq q,
ã = Wa a, ṽ = Wv v, and ṽ0 = Wv0 v. For simplicity, it is
assumed that both VQA and VQG adopt v as visual input,
which can be replaced by the post-attention feature va or
vq . Noticing both Tc and Tc0 model the interactions among
the image, question, and answer embeddings, but with different dimension arrangement, we assume the relationship
between Tc0 and Tc as follows:
Tc0 [:, i, :] = Tc> [:, i, :]

(12)

Additionally, the transform matrices for visual information
Wv0 and Wv can also be shared. Therefore, we can unify
the question and answer embedding inference with single
three-way operator Tc :
ã∗ = (Tc ×1 q̃) ×2 ṽ
q̃∗ = (Tc ×3 ã) ×2 ṽ

(13)

Furthermore, since Tc [:, i, :] represents the correlation between the re-parameterized question and answer embeddings, considering the duality of Q and A, we could assume
the the following for Tc [:, i, :]:
(
ta = tq = t
(14)
Tc [:, i, :] = Tc> [:, i, :], i ∈ [1, tv ]
Correspondingly, Eq. 13 could be written as:
ã∗ = (Tc ×1 q̃) ×2 ṽ
q̃∗ = (Tc ×1 ã) ×2 ṽ

(15)

That is to say, we could infer ã or q̃ by just alternating the
mode-1 input of the kernel.
By introducing the sparsity constraint like Eq. 8, the inference of answer and question features ã∗ and q̃∗ can be
reformulated as:

Consequently, given a question/answer pair (q, a), the predicted answer/question representations are expected to have
the following form:
a ≈ â = φ(q, v) and q ≈ q̂ = φ∗ (a, v).

(18)

(16)

To leverage the property above, we propose the Duality Regularizer, smoothL1 (q̂ − q) and smoothL1 (â − a), where
the loss function smoothL1 is defined as:

0.5 ∗ x2 ,
if |x| < 1
smoothL1 (x) =
(19)
|x| − 0.5, otherwise

And the target answer and question embeddings are provided by:
â = ã∗> × Wa
(17)
q̂ = q̃∗> × Wq

By minimizing Q/A duality loss, primal and dual question/answer representations are unified, and VQG and VQA
are linked with each other. Moreover, the Duality Regularizer can be viewed as a way to provide soft targets for the
question/answer feature learning.

∗

ã =

R
X

>



>



q̃ Mr

>



>



ṽ Nr

r=1
∗

q̃ =

R
X

ã Mr

ṽ Nr

r=1

As shown in Fig. 3, we unify the two Mutan modules by
sharing parameters Wa , Wq , Wv , and Tc . And we call this
invertible module Dual Mutan.
Furthermore, when the decoder after the dual Mutan are
considered, the predicted answer embedding â will be fed
into another linear transform layer to get the per-class score,
and the question embedding q̂ will be decoded by LSTM,
both of which have linear transforms afterwards. So the linear transforms in Eq. 17 can be skipped for efficiency. And
we can directly use ã∗ and q̃∗ as the predicted features to
feed into decoders.

Dual Training

Parameter Sharing for Q and A encoding and
decoding.

where L(vqa) (a, a∗ ) and L(vqg) (q, q ∗ ) adopt the multinomial classification loss (Ben-younes et al. 2017) and sequence generation loss (Vinyals et al. 2015) as the unary loss
for VQA and VQG components respectively.
Additionally, as every operation is differentiable, the entire model can be trained in an end-to-end manner. In the
next section, we will show that our dual training strategy
could bring significant improvement for both VQA and
VQG models.

Considering the duality of VQA and VQG, the encoder and
decoder of Q/A can be viewed as reverse transformation to
each other. Hence, we could employ these properties to propose corresponding weight sharing scheme.
For input answers in the VQG component, the input answer is embedded into features a by the matrix Ea , which
stores the embeddings of each answer. For the answer generation in the VQA component, the predicted feature â is
decoded for obtaining the answer through a linear classifier Wa , which can be regarded as a set of per-class templates for the feature matching. Thus, we can directly share
the weights of Ea and Wa , where Ea is required to be the
transposition of Wa .
For input questions in the VQA component, LSTM is applied to encode the question sentence into a fixed-size feature vector q. For the question generation in the VQG component, LSTM is also applied to decode the vector back to
a word sequence step-by-step. We also share the parameters
of the two LSTMs. Experimental results show that sharing
weights of two LSTMs will not deteriorate the final result
but requires fewer parameters.

Duality Regularizer
With Dual Mutan, we have reformulated the feature fusion
part of VQA and VQG (φ and φ∗ ) as the inverse process to
each other. φ and φ∗ are expected to form a closed cycle.

With our proposed weight sharing schema (Dual Mutan and
Sharing De-/Encoder), our VQA and VQG models can be
reconstructed to inverse process of each other with sharing
parameters. Hence, joint training on VQG and VQA tasks
introduces the invertibility of the model as an additional regular term to regulate the training process. The overall training loss including our proposed Q/A duality is as below:
Loss =L(vqa) (a, a∗ ) + L(vqg) (q, q ∗ )
+ smoothL1 (q − q̂) + smoothL1 (a − â)

(20)

Experiments
Model implementation details, data preparation and experiment results will be shown in this section. Qualitative results
will be shown in the supplementary materials.

Implementation Details
Our iQAN is based on PyTorch implementation of Mutan VQA (Ben-younes et al. 2017). We directly use the
ImageNet-pretrained ResNet-152 (He et al. 2015) provided
by PyTorch as our base model and keep this part fixed. All
images are resized to 448×448, and the size of feature maps
is 14 × 14. Newly introduced parameters are randomly initialized. Adam (Kingma and Ba 2015) with fixed learning
rate 0.0001 is used to update the parameters. The training
batch size is 512.2 All models are trained for 50 epochs, and
the best validation results are used as final results.
2
Batch size will influence the model performance. To be fair,
we use 512 for all experiments.

model Dual Mutan
1
2
3
4
5

Duality
Sharing
Regularizer De- & Encoder

X
X
X
X

X
X

X
X

VQA2 Subset
VQA2 Full
(Goyal et al. 2017) (Goyal et al. 2017)
acc@1 acc@5
acc@1 acc@5
50.72
50.99
51.23
51.38
51.49

78.56
78.71
78.80
78.96
78.93

54.83
54.78
54.70
54.39
54.97

87.68
87.66
88.07
87.92
87.77

size
86.76
87.29
87.42
87.75
87.75

CLEVR
(Johnson et al. 2017)
material shape color
88.25
88.10
88.81
88.48
88.91

82.26
82.82
82.84
84.28
84.08

76.86
77.62
77.73
77.97
78.86

Overall
83.74
84.13
84.40
84.78
85.07

Table 1: Ablation study of different settings. Dual Mutan: our proposed sharing Mutan scheme. Duality Regularizer: an
additional regular term defined in Eq. (19) and (20) to guarantee the similarity of dual pairs (q ≈ q̂ and a ≈ â). Sharing De& Encoder: parameter sharing scheme for decoders and encoders of Q and A. Model 1 is the baseline model with separated
VQA and VQG models. Additionally, the per-question-type top-1 accuracies on CLEVR are also listed.
Train
Validation
#images #Q,A pairs #images #Q,A pairs
VQA2 163,550
68,434
78,047
33,645
CLEVR 107,132
57,656
22,759
12,365
Dataset

Table 2: Statistics of filtered VQA2 (Goyal et al. 2017) and
CLEVR (Johnson et al. 2017).

Data Preparation
We evaluate the proposed method on two large-scale VQA
datasets, VQA2 (Goyal et al. 2017) and CLEVR (Johnson et
al. 2017), both of which provide images and labeled hQ,Ai
pairs. However, these two datasets contain some of the questions with non-informative answers as yes/no or number. It
is nearly impossible for a model to generate expected questions from an answer like yes. Therefore, we preprocess the
data to filter out some question-answer pairs for both the
VQA2 and the CLEVR to fairly explore the duality of Q
and A: For VQA2, we only select the questions whose annotated question type starts with “what”, “where” or “who”.
For CLEVR, the questions starting with “what” and whose
answer is not number are selected. Additionally, for VQA2,
we fixed the answer vocabulary size to 2000 most frequent
answers as in (Ben-younes et al. 2017). hQ,Ai pairs whose
answer is not in the vocabulary will be removed. Detailed
statistics of filtered VQA2 and CLEVR are shown in Table 2.

on different settings. Model 1 is the baseline model with separated VQA and VQG models.
First, we focus on the filtered VQA2 dataset. By comparing model 1 and 2, we can see that our proposed Dual
Mutan can help to improve VQA but not significantly. Just
as we discussed below, the derivation of Dual Mutan module is based on a lot of assumptions guaranteed by the duality regularizer and the encoder/decoder sharing. When further adding these two components, the model performance is
continuously improved, and the full model shown in 5 outperforms the baseline model by 0.77% on top-1 accuracy,
which is a significant improvement for VQA.
Furthermore, similar experiments are done on the full
VQA2 dataset, but there is little improvement, while model
2∼4 are even worse than the baseline model. That is mainly
because generating expected questions from answers like
yes or no is almost impossible, where the information provided by answers is too little for question generation. Even
worse, VQG loss will dominate the model training, which
may deteriorate the VQA performance.
We also evaluate our proposed method on the CLEVR
dataset, which is designed to diagnose the reasoning ability
of VQA models. By comparing our full model and baseline
model, we can see that our dual training scheme could help
to improve the reasoning ability of the VQA model (1.33%
gain on overall Acc@1). In addition, since VQA and VQG
model are inverse form of each other, the dual training of
VQG and VQA can be regarded as training a model to understand the question and then ask a similar one, so the model
gets more training on reasoning ability.

Performance Metrics
VQA is commonly formulated as the multinomial classification problem while VQG is a sequence generation problem. Therefore, we use top-1 accuracy (Acc@1) and top-5
accuracy (Acc@5) to measure the quality of the predicted
answers. Sentence-level BLEU score (Papineni et al. 2002)
provided by NLTK (Bird 2006) is employed to evaluate the
generated questions (with Method 4 smooth function).

Component Analysis
We compare our proposed Dual Training scheme with the
baseline Mutan model on three datasets, filtered VQA2, full
VQA2 and filtered CLEVR. Table 1 shows our investigation

Dual Learning for Other VQA Models
Our proposed dual training mechanism can be viewed as reconstructing VQA model to finish VQG problem. By sharing parameters, the model is trained with two tasks. Although the dual training method is derived from Mutan, but
the core idea can also be applied to other latest VQA models (Zhou et al. 2015; Kim et al. 2017) (shown in Table 3).
iBOWIMG is a simple baseline bag-of-word (BOW) VQA
model which simply concatenates image and question embeddings to predict the answer. Correspondingly, we implement a dual VQG model with similar feature fusion. Since
there is no parameters for fusion part, Dual Training only

Model
Baseline
Dual Training
Gain

iBOWIMG (Zhou et al. 2015)
MLB (Kim et al. 2017)
MUTAN + SkipThought
MUTAN + LSTM
Acc@1 Acc@5
BLEU
Acc@1 Acc@5 BLEU Acc@1 Acc@5 BLEU Acc@1 Acc@5 BLEU
42.05
43.44
1.39

72.79
74.27
1.48

55.23
55.36
0.13

50.23
50.83
0.60

77.64
78.12
0.48

55.35
55.60
0.25

50.72
51.49
0.77

78.56
78.93
0.37

54.15
54.83
0.68

49.91
50.78
0.87

77.47
78.16
0.69

54.17
54.89
0.72

Table 3: Evaluation of Dual Training Scheme on different VQA models. Acc@1 and Acc@5 are the VQA metrics, while
BLEU score is used to measure the question generation quality. Baseline models are separately-trained VQA and VQG. Dual
Training is to employ our proposed parameter sharing schemes and Dual Regularizer. The Dual Training version is to train one
model with two tasks while Baseline is to train two different models. SkipThought and LSTM denote two question encoders
used in the model.
requires decoder & encoder sharing and duality regularizer.
Experiment results show that jointly training VQG and VQA
could bring mutual improvements to both, especially for
VQA model (1.39% on Acc@1). However, the improvement
for VQG is not significant, because iBOWIMG VQA uses
BOW to encode questions while the VQG model uses LSTM
to decode question features. Therefore, the compulsive similarity of predicted features for LSTM and BOW-encoded
feature will be too strong as a regularizer. In addition, generating questions is hard and the baseline VQG performance
is already high when comparing with other models, so there
is little room for improvement.
MLB is another latest bilinear VQA model that can be
viewed as the special case of Mutan which sets the core bilinear operator T c to identity. Therefore, the derived dual
training scheme can be applied to MLB model directly and
it can help VQG and VQA improve each other.
Mutan + X: The original Mutan model in (Ben-younes et
al. 2017) utilizes the pretrained skip-thought model (Kiros
et al. 2015) as question encoder, so we change that to
LSTM (trained from scratch) to make it sharable with the
decoder. For both versions, the dual training could consistently bring gains to VQA and VQG. Besides, by comparing two versions, we could find that the pretrained encoder
performs better on VQA, which is a trick to improve VQA
performance while hardly influencing VQG.
By applying the dual training on three latest models, we
can see that even though our proposed method is derived
from Mutan, it can be generalized to other VQA models and
bring concordant improvements.

Augmenting VQA with VQG
Since VQG could provide more questions given answers,
besides as a dual task to train VQA model, VQG could
also help to generate expensively-labeled questions from
cheaply-labeled concepts (answers) to produce more training data with little cost. So we propose two ways to employ
the augmented data by VQG and evaluate them on filtered
VQA2 dataset. In this section, the training set will be partitioned into two parts: one is with hQ,Ai pairs (Set 1); the
other only contains answers (Set 2). Experiment results are
shown in Table 4.
VQG+X: We first train a VQG model (with dual training)
with on Set 1, and use it to generate questions given the answers in Set 2. Then we combine the Set 1 and augmented

Model

Dataset

Baseline
DT
VQG+Baseline
VQG+DT
VQG+DT+FT

0.5 hQ,Ai
0.5 hQ,Ai
0.5 hQ,Ai + 0.5 A
0.5 hQ,Ai + 0.5 A
0.5 hQ,Ai + 0.5 A

Acc@1 Acc@5 BLEU
46.68
47.63
47.51
47.99
48.48

74.43
75.42
75.39
75.79
76.23

50.96
53.23
45.70
46.06
53.78

Baseline
DT
VQG+Baseline
VQG+DT
VQG+DT+FT

0.1 hQ,Ai
0.1 hQ,Ai
0.1 hQ,Ai + 0.9 A
0.1 hQ,Ai + 0.9 A
0.1 hQ,Ai + 0.9 A

33.60
35.23
37.83
38.87
39.95

61.04
62.77
64.86
66.02
66.67

47.26
48.45
44.90
44.92
49.40

Table 4: Our investigation on augmenting hQ,Ai pairs using VQG with A given. Baseline is separately trained VQA
and VQG models. DT denotes the dual training. 0.1 and 0.5
denote the proportion of training data used as Set 1.
Set 2 as training data. X can be baseline or dual trained (DT)
model. Results show that, compared to the model trained
only on Set 1, such method will improve VQA but deteriorate VQG performance. It is mainly because the generated
questions are not identical to the original ones, as one answer could correspond to several reasonable questions. So
the generated questions may follow a different distribution.
Hence, learning to generate questions from Set 2 will deteriorate the performance of VQG on the validation set. On the
other hand, most of the generated questions can be answered
by the given answer, which is the reason why they can serve
as the augmented data to boost the VQA performance.
VQG+DT+FT: Although Set 2 provides extra training
data, the quality is not so good as Set 1. Therefore, a better
way is to pretrain using dual training (DT) for the model on
Set 1 and augmented Set 2, and then finetune (FT) the model
on Set 1. From experiment results, we can see that our proposed data augmentation method significantly outperforms
the vanilla dual-trained models (VQG+DT+FT v.s. DT), our
proposed method could successfully leverage the additional
annotated answers to boost the model training.

Conclusion
We present the first attempt to consider visual question generation as a dual task of visual question answering. Corre-

spondingly, we proposed a dual training scheme, iQAN, that
is derived from Mutan VQA model but also applied to some
other latest VQA models. Our proposed method could reconstruct VQA model to VQG and train a single model with
two conjugate tasks. Experiments show that our dual trained
model outperforms the baseline model on both VQA2 and
CLEVR dataset, and it consistently brings gains to several
latest VQA models. We further investigate the potential of
using VQG to augment training data. Our proposed method
is proved to be an effective way to leverage the cheaplylabeled answers to boost the VQA and VQG models.

Acknowledgment
This work is supported by Hong Kong Ph.D. Fellowship
Scheme, SenseTime Group Limited and Microsoft Research
Asia. We also thank Duyu Tang, Yeyun Gong, Zhao Yan,
Junwei Bao and Lei Ji for helpful discussions.

References
Ali, H.; Chali, Y.; and Hasan, S. A. 2010. Automation of question
generation from sentences. In Proceedings of QG2010: The Third
Workshop on Question Generation.
Antol, S.; Agrawal, A.; Lu, J.; Mitchell, M.; Batra, D.;
Lawrence Zitnick, C.; and Parikh, D. 2015. Vqa: Visual question
answering.
Ben-younes, H.; Cadene, R.; Cord, M.; and Thome, N. 2017.
Mutan: Multimodal tucker fusion for visual question answering.
ICCV.
Bird, S. 2006. Nltk: the natural language toolkit. In ACL.
Fukui, A.; Park, D. H.; Yang, D.; Rohrbach, A.; Darrell, T.; and
Rohrbach, M. 2016. Multimodal compact bilinear pooling for
visual question answering and visual grounding. EMNLP.
Goyal, Y.; Khot, T.; Summers-Stay, D.; Batra, D.; and Parikh, D.
2017. Making the v in vqa matter: Elevating the role of image
understanding in Visual Question Answering. In CVPR.
He, K.; Zhang, X.; Ren, S.; and Sun, J. 2015. Deep residual learning for image recognition. arXiv preprint arXiv:1512.03385.
He, D.; Xia, Y.; Qin, T.; Wang, L.; Yu, N.; Liu, T.; and Ma, W.-Y.
2016. Dual learning for machine translation. In NIPS.
Jain, U.; Zhang, Z.; and Schwing, A. 2017. Creativity: Generating
diverse questions using variational autoencoders. arXiv preprint
arXiv:1704.03493.
Johnson, J.; Hariharan, B.; van der Maaten, L.; Fei-Fei, L.; Zitnick,
C. L.; and Girshick, R. 2017. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. CVPR.
Johnson, J.; Karpathy, A.; and Fei-Fei, L. 2015. Densecap: Fully
convolutional localization networks for dense captioning. arXiv
preprint arXiv:1511.07571.
Kalady, S.; Elikkottil, A.; and Das, R. 2010. Natural language
question generation using syntax and keywords. In Proceedings of
QG2010: The Third Workshop on Question Generation.
Kim, J.-H.; On, K.-W.; Kim, J.; Ha, J.-W.; and Zhang, B.-T. 2017.
Hadamard product for low-rank bilinear pooling. ICLR.
Kingma, D., and Ba, J. 2015. Adam: A method for stochastic
optimization. ICLR.
Kiros, R.; Zhu, Y.; Salakhutdinov, R. R.; Zemel, R.; Urtasun, R.;
Torralba, A.; and Fidler, S. 2015. Skip-thought vectors. In NIPS.

Lu, J.; Yang, J.; Batra, D.; and Parikh, D. 2016. Hierarchical question-image co-attention for visual question answering. In
NIPS.
Malinowski, M., and Fritz, M. 2014. Towards a visual turing challenge. NIPS workshop.
Malinowski, M.; Rohrbach, M.; and Fritz, M. 2015. Ask your
neurons: A neural-based approach to answering questions about
images. In ICCV.
Manning, C. D.; Schütze, H.; et al. 1999. Foundations of statistical
natural language processing. MIT Press.
Martin, J. H., and Jurafsky, D. 2000. Speech and language processing.
Mora, I. M.; de la Puente, S. P.; and Giro-i Nieto, X. 2016. Towards automatic generation of question answer pairs from images.
CVPRW.
Mostafazadeh, N.; Misra, I.; Devlin, J.; Mitchell, M.; He, X.; and
Vanderwende, L. 2016. Generating natural questions about an
image. arXiv preprint arXiv:1603.06059.
Papineni, K.; Roukos, S.; Ward, T.; and Zhu, W.-J. 2002. Bleu: a
method for automatic evaluation of machine translation. In ACL.
Serban, I. V.; Garcı́a-Durán, A.; Gulcehre, C.; Ahn, S.; Chandar, S.;
Courville, A.; and Bengio, Y. 2016. Generating factoid questions
with recurrent neural networks: The 30m factoid question-answer
corpus. arXiv preprint arXiv:1603.06807.
Shih, K. J.; Singh, S.; and Hoiem, D. 2016. Where to look: Focus
regions for visual question answering. In CVPR.
Sundaram, N.; Brox, T.; and Keutzer, K. 2010. Dense point trajectories by gpu-accelerated large displacement optical flow. In
ECCV.
Tang, D.; Duan, N.; Qin, T.; and Zhou, M. 2017. Question answering and question generation as dual tasks. arXiv preprint
arXiv:1706.02027.
Vinyals, O.; Toshev, A.; Bengio, S.; and Erhan, D. 2015. Show and
tell: A neural image caption generator. In CVPR.
Yu, D.; Fu, J.; Mei, T.; and Rui, Y. 2017. Multi-level attention
networks for visual question answering. In CVPR.
Zhang, S.; Qu, L.; You, S.; Yang, Z.; and Zhang, J. 2016. Automatic generation of grounded visual questions. arXiv preprint
arXiv:1612.06530.
Zhou, B.; Tian, Y.; Sukhbaatar, S.; Szlam, A.; and Fergus, R. 2015.
Simple baseline for visual question answering. arXiv preprint
arXiv:1512.02167.
Zhu, J.-Y.; Park, T.; Isola, P.; and Efros, A. A. 2017. Unpaired
image-to-image translation using cycle-consistent adversarial networks. ICCV.

Supplementary Materials: Qualitative Results
cyan

Question answering and question generation results on VQA2 validation set are shown in this section. Both VQA and VQG are
jointly trained with our proposed dual training scheme, iQAN.

Q: where is the microwave
A: above stove
----------------------------------A: above stove
Q: where is the microwave

Q: what kind of animals are shown
A: zebras
---------------------------------------------A : zebras
Q : what animals are these

Q: what is on the pizza
A: vegetables
----------------------------------A: asparagus
Q: what is the green vegetable

Q: what flag is flying on this
building
A: above stove
--------------------------------A: above stove
Q: where is the microwave

Q: what color is the background
A: blue
----------------------------------A: blue
Q: what color is the sky

Q: what type clouds are
in the photo
A: cumulus
----------------------------A: cumulus
Q: what type of clouds
are in the sky

Q: what is the plane flying over
A: mountains
----------------------------------------A: mountains
Q: what is in the background

Q: what game is being played
A: baseball
----------------------------------A: baseball
Q: what sport is this

Q: what are the toppings
A: cheese
------------------------------------------A: broccoli
Q: what is the green vegetable on the pizza

Q: what animal is on his shirt
A: dog
----------------------------------------A: lion
Q: what animal is on the man 's shirt

Q: what brand vehicle is in the foreground
A: ford
----------------------------------A: nissan
Q: what brand is the truck

Q: what are the giraffes
reaching for
A: food
---------------------------A: leaves
Q: what are the giraffes
eating

Q: what is the yellow stuff
on the hot dogs
A: mustard
---------------------------A: mustard
Q: what is the red stuff on
the hot dog

Figure 4: Qualitative results of dual-trained VQA (top) and VQG (bottom) models. Labeled questions/answers are given to
infer the counterpart. Red denotes the failure cases. Green indicates the correct ones. The question is viewed as correct if it
corresponds to the given answer.

Analysis
From the qualitative results, we can see that, our VQA and VQG models have the ability to recognize the image, understand
the questions, and even know some uncommon words (like cumulus). The VQA model can associate the question and image to
find the answer, while VQG model can generate questions corresponding to the given answers although they are not identical
to the labeled ones. Additionally, some typical failure cases are shown, including generating the wrong questions (that are
not corresponding the given answers), giving the wrong answer due to poor visual recognition ability or providing another
reasonable answer but different from the label. Sometimes, questions are too hard to answer, e.g. to recognize the brand of
vehicles, where the question involves fine-grained categories and tiny objects. But overall, the generated questions are similar
to the human-annotated ones and correspond to the answers, which explains why they could serve as augmented data to improve
VQA performance.

