arXiv:1802.03803v1 [] 11 Feb 2018

F LIP D IAL: A Generative Model for Two-Way Visual Dialogue
Daniela Massiceti
University of Oxford, UK

N. Siddharth
University of Oxford, UK

Puneet Kumar Dokania
University of Oxford, UK

daniela@robots.ox.ac.uk

nsid@robots.ox.ac.uk

puneet@robots.ox.ac.uk

Philip H.S. Torr
University of Oxford, UK
phst@robots.ox.ac.uk
Question

Ground-truth
answer

How old is the
I can’t see his
man?
face, but maybe mid
or late twenties
What race is
the man?

White

Z1

Z2

Z3

Z4

He looks
thirty

Twenties

I can’t
tell

White

White

In fivethirty
away

Z5

Thirtys

z6

I say
teenager

Twentys

I can’t Caucasian
see

He looks
white

Is he wearing
a hat?

I don’t see his
head

No

Yes

Yes

Yes

No

Is he wearing
a shirt?

Yes

Yes

Yes

No

No

Yes

It

Dark grey

Grey

Grey

Black

White

Black

Black

Boy

I

Male

I

I

What color?
What gender is
the baby?

I can’t really
tell, maybe a girl

I

What is the
baby wearing?

A bib

Shirt

What color is
the remote?

White and black

White

Silver

No

No

Is the chair
wood?
What color is
the chair?

No, it’s leather
Like a light
burgundy

T-shirt Shirt and Shirt and
has pants
shirt
shirt

Brown White has
white
checkered

Silver

It is
white and
black

Yes No it’s a
chair
A light
brown

Gray

No

Looks Shirt and
like
shirt
white
White

Black

No

Yes

Brown
with
white
texture

Gray

A man sitting in a chair holding a baby who is chewing on a remote

Figure 1: Diverse answers generated by F LIP D IAL in the one-way visual dialogue (1 VD) task. For a given time step (row),
each column shows a generated answer to the current question. Answers are obtained by decoding a latent zi sampled from
the conditional prior – with conditions being image, caption and dialogue history up until that time step.

Abstract

1. Introduction
A fundamental characteristic of a good human-computer
interaction (HCI) system is its ability to effectively acquire
and disseminate knowledge about the tasks and environments in which it is involved. A particular subclass of such
systems, natural-language-driven conversational agents such
as Alexa and Siri, have seen great success in a number of welldefined language-driven tasks. Even such widely adopted
systems suffer, however, when exposed to less circumscribed,
more free-form situations. Ultimately, an implicit requirement for the wide-scale success of such systems is the effective understanding of the environments and goals of the user
– an exceedingly difficult problem in the general case as it
involves getting to grip with a variety of sub-problems (semantics, grounding, long-range dependencies) each of which
are extremely difficult problems in themselves. One avenue
to ameliorate such issues is the incorporation of visual context to help explicitly ground the language used – providing
a domain in which knowledge can be anchored and extracted
from. Conversely, this also provides a way in which language
can be used to characterise visual information in richer terms,

We present F LIP D IAL, a generative model for Visual
Dialogue that simultaneously plays the role of both participants in a visually-grounded dialogue. Given context in the
form of an image and an associated caption summarising
the contents of the image, F LIP D IAL learns both to answer
questions and put forward questions, capable of generating
entire sequences of dialogue (question-answer pairs) which
are diverse and relevant to the image. To do this, F LIP D IAL
relies on a simple but surprisingly powerful idea: it uses
convolutional neural networks (CNNs) to encode entire dialogues directly, implicitly capturing dialogue context, and
conditional VAEs to learn the generative model. F LIP D IAL
outperforms the state-of-the-art baseline in the sequential
answering task (1 VD) on the VisDial dataset by a significant
margin of 12 points in Mean Rank. We are the first to extend
this paradigm to full two-way visual dialogue (2 VD), where
our model is capable of generating visually-grounded both
questions and answers in sequence, for which we propose a
set of novel evaluation measures and metrics.

1

for example with sentences describing salient features in the
image (referred to as “captioning”) [12, 14].
In recent years, there has been considerable interest in
visually-guided language generation in the form of visual
question-answering (VQA) [1], and subsequently visual dialogue [5], demonstrating the ability to answer questions
in the context of an image. In the particular case of visual
dialogue, along with the image, previously seen questions
and answers (i.e. the dialogue history) are also accepted, and
a relevant answer at the current time produced. We refer
to this one-sided or answer-only form of visual dialogue as
one-way visual dialogue (1 VD). Inspired by these models
and aiming to extend their capabilities, we establish the task
of two-way visual dialogue (2 VD) whereby an agent must
be capable of acting as both the questioner and the answerer.
Our motivation for this is simple – AI agents need to
be able to both ask questions and answer them, often interchangeably, rather do either one exclusively. For example,
a vision-based home-assistant robot (e.g. Amazon’s Alexa
on legs) may need to ask questions based on her visual input
(“There is no toilet paper in the cupboard. Would you like
me to order more for you?”) but may also need to answer
questions asked by humans (“Is the brand of toilet paper
you’ve ordered the one with the puppies on it?”). The same
question-and-answer capability is true for other applications.
For example, in the case of aids for the visually-impaired, a
user may need the answer to “Where is the tea and kettle?”,
but the system may equally need to query “Do you want Earl
Grey or a Rooibos tea?” to resolve potential ambiguities.
We take one step toward this broad research goal with
F LIP D IAL, a generalisable generative model capable of both
1 VD and 2 VD. The generative aspect of our model is served
by using the conditional variational auto-encoder (CVAE), a
framework for learning deep conditional generative models
while simultaneously amortising the cost of inference in such
models over the dataset [16, 24]. Furthermore, inspired by
the recent success of convolutional neural networks (CNNs)
in language generation and prediction tasks [10, 13, 21], we
explore the use of CNNs on sequences of sequences (i.e. a
dialogue) to implicitly capture all sequential dependences
through the model. Demonstrating the surprising effectiveness of this approach, we show sets of sensible, visuallygrounded answer generations for the 1 VD task from our
F LIP D IAL model in Fig. 1.
We here provide a brief treatment of works related to
visual dialogue. We reserve a thorough comparison to Das
et.al. [5] for Section 4.5, but note that our fully-generative
convolutional extension of their model significantly outperforms their state-of-the-art results on the answering of sequential visual-based questions (1 VD). Another work, Das
et.al. [6] present a Reinforcement Learning based model to
do 1 VD, where they instantiate two separate agents, one each
for questioning and answering. Crucially, the two agents

are given different information – with one (QBot) given the
caption, and the other (ABot) given the image. While this
sets up the interesting task of performing image retrieval
from natural-language descriptions, it is also fundamentally
different from having a single agent perform both roles. Jain
et.al. [11] explore a complementary task to VQA [1] where
the goal is instead to generate a (diverse) set of relevant
questions given an image. In their case, however, there
is no dependence on a history of questions and answers.
Moreover, the evaluations conducted are characteristic to the
dataset employed (VQG [18]). Finally, we note that Zhao
et.al. [27] employ a similar model structure to ours, using
a CVAE to model dialogue, but condition their model on
discourse-based constraints for a purely linguistic (rather
than visuo-linguistic) dataset. The tasks we target, our architectural differences (CNNs instead of recurrent networks),
and the dataset and metrics we employ are distinct.
Our primary contributions in this work are therefore:
• A generalisable, fully-generative, convolutional framework for visual dialogue that outperforms state-of-the-art
models on sequential question answering (1 VD), and establishes the first baseline in the challenging two-way visual
dialogue task (2 VD).
• Realistic evaluation settings using the predicted, rather
than ground-truth dialogue, histories – essential for realworld conversational agents.
• Novel evaluation metrics for generative models of twoway visual dialogue to quantify answer-generation quality,
question relevance, and the models’s generative capacity.

2. Preliminaries
Here we present a brief treatment of the preliminaries for
deep generative models – a conglomerate of deep neural networks and generative models. In particular, we discuss the
variational auto-encoder (VAE) [16] which given a dataset X
with elements x ∈ X , simultaneously learns i) a variational
approximation qφ (z | x)1 to the unknown posterior distribution pθ (z | x) for latent variable z, and ii) a generative
model pθ (x, z) over data and latent variables. These are
both highly attractive prospects as the ability to approximate
the posterior distribution helps amortise inference for any
given data point x over the entire dataset X , and learning
a generative model helps effectively capture the underlying
abstractions in the data. Learning in this model is achieved
through a unified objective, involving the marginal likelihood
(or evidence) of the data, namely:
log pθ (x) = DKL (qφ (z | x) k pθ (z | x))
+ Eqφ (z|x) [log pθ (x, z) − log qφ (z | x)]
≥ Eqφ (z|x) [log pθ (x, z) − log qφ (z | x)] (1)
1 Following the literature, the terms recognition model or inference
network may also be used to refer to the posterior variational approximation.

The intractability of computing the unknown true posterior pθ (z | x) in the Kullback-Leibler (KL) divergence
makes it difficult to optimise directly. Since the KL divergence is always ≥ 0, however, the second term lower-bounds
the marginal log-likelihood log pθ (x), which we can adopt
as a surrogate to optimise instead. This surrogate objective
function is referred to as the evidence lower bound (ELBO).
This model can then be extended by introducing a conditioning variable y to capture a conditional posterior approximation qφ (z | x, y) and a conditional generative model
pθ (x, z | y), deriving the CVAE [24]. Similar to Eq. (1), it
defines as its objective the conditional ELBO:
log pθ (x | y)
≥ Eqφ (z|x,y) [log pθ (x, z | y) − log qφ (z | x, y)]
= Eqφ (z|x,y) [log pθ (x | z, y)]
− DKL (qφ (z | x, y) k pθ (z | y))

(2)

where the first term is referred to as the reconstruction or
cross entropy (CE) term, and the second, the regularisation
or KL divergence term. Here too, similar to the VAE, the
variational approximation qφ (z | x, y) and the generative
model prior pθ (z | y) are typically taken to be isotropic multivariate Gaussian distributions, whose parameters (µq , σq2 )
and (µp , σp2 ) are provided by deep neural networks (DNNs)
with parameters φ and θ, respectively. The generative model
likelihood pθ (x | z, y), whose form varies depending on the
data type – Gaussian or Laplace for images and Categorical
for language models – is also parametrised similarly. In this
work, we employ the CVAE model as it lends itself naturally
to the task of eliciting dialogue given contextual information
from vision (images) and language (captions).

3. Generative Models for Visual Dialogue
In applying deep generative models to visual dialogue, we
begin by characterising a preliminary step toward it, VQA. In
VQA, the goal is to answer a single question in the context of
a visual cue, typically an image. The primary goal for such
a model is to ensure that the elicited answer conforms to a
stronger notion of relevance than simply answering the given
question – it must also relate to the visual cue provided. This
notion can be extended to one-way visual dialogue (1 VD)
which we define as the task of answering a sequence of
questions contextualised by an image (and a short caption
describing its contents), similar to [5]. A generative approach to 1 VD can be adopted by introducing stochasticity
in the latent representations. Being able to exclusively answer questions, however, is not fully encompassing of true
conversational agents. We therefore extend 1 VD to the more
general and realistic task of two-way visual dialogue (2 VD)
by enforcing that the model elicit not just answers given
questions, but questions given answers as well – generating

both components of a dialogue, contextualised by the given
image and associated caption. As with 1 VD, one can also
formalise the task of 2 VD within a generative framework.
As such, we now characterise our generative approach
to 2 VD in the CVAE framework. For a given image i and
associated caption c, we define a dialogue as a sequence
T
of question-answer pairs d1:T = h(qt , at )it=1 , simply denoted d where sequence indexing is unnecessary. Additionally, we denote a dialogue context h, for when there exists
an explicit dependency between the dialogue history and the
currently elicited question or answer. When indexed by step
as ht , it captures the dialogue subsequence d1:t .
With this formalisation, we characterise a generative
model for 2 VD under latent variable z as pθ (d, z | i, c, h) =
pθ (d | z, i, c, h) pθ (z | i, c, h), with the corresponding
recognition model defined as qφ (z | d, i, c, h). Note that
with relation to Eq. (2), data x is dialogue d and conditioning
variable is y = {i, c, h}, giving:
log pθ (d | i, c, h)
≥ Eqφ (z|d,i,c,h) [log pθ (d | z, i, c, h)]
− DKL (qφ (z | d, i, c, h) k pθ (z | i, c, h)),

(3)

with the graphical model structures shown in Fig. 2.
i

c

h
z

c

i
z

d

h

d

Figure 2: Left: Conditional recognition model and Right:
conditional generative model for 2 VD.
We show that the formulation in Eq. (3) is general enough
to be applied to the different levels of visually-guided language models, from single question-answering (VQA) all the
way to full two-way dialogue generation (2 VD). Taking a
step back from generative 2 VD, we can re-frame the formulation for generative 1 VD (i.e. sequential answer generation)
by considering the generated component to be the answer to
a particular question at step t, given context from the image,
caption and the sequence of previous question-answer pairs.
Simply put, in the formulation of Eq. (2), this corresponds
to the data x being simply the answer at , conditioned on the
image, its caption, the dialogue history to t-1, and the current
question, or y = {i, c, ht−1 , qt }. For notational simplicity, we denote a compound context as h+t = hht−1 , qt i and
reformulate Eq. (3) for 1 VD as:
log pθ (d | i, c, h) =
log pθ (at | i, c, h+t )

T
X

log pθ (at | i, c, h+t )

t=1

≥ Eqφ (z|at ,i,c,h+t ) [log pθ (at | z, i, c, h+t )]
− DKL (qφ (z | at , i, c, h+t ) k pθ (z | i, c, h+t )),

(4)

i
c

h+t
z

h+t
at
T

z

at

i
c

T

Figure 3: Left: Conditional recognition model and Right:
conditional generative model for 1 VD.
with the graphical model structures shown in Fig. 3.
Our baseline [5] for the 1 VD model can also be represented in our formulation by taking the variational posterior
and generative prior to be conditional Dirac-Delta distributions. That is, qφ (z | at , i, c, h+t ) = pθ (z | i, c, h+t ) =
δ(z | i, c, h+t ). This transforms the objective from Eq. (4)
by a) replacing the expectation of the log likelihood over
the recognition model by an evaluation of the log likelihood
for a single encoding (one that satisfies the Dirac-Delta),
and b) ignoring DKL regulariser, which is trivially 0, computing the marginal likelihood directly as just the model
likelihood log pθ (at | z, i, c, h+t ), where z ∼ δ(z | i, c, h+t ).
Note that while such models can “generate” answers to
questions by sampling from the likelihood function, we typically don’t call them generative since they effectively make
the encoding of the data and conditions fully deterministic.
We explore and demonstrate the benefit of a fully-generative
treatment of 1 VD in Section 4.5. It also follows trivially that
the basic building-block in this setup, the VQA model (for
single question-answering), itself can be obtained from this
model for 1 VD by simply assuming there is no accumulation
of dialogue history (i.e. step length (T = 1).

3.1. “Colouring” Visual Dialogue with Convolutions
The convolutional formulation of F LIP D IAL allows us
to implicitly capture the sequential nature of sentences (i.e.
questions and answers) as well as sequences of sentences
(i.e. whole dialogues). Here we introduce how questions,
answers, and subsequently whole dialogues are represented
in a way that allows them to be encoded by convolutional
neural networks (CNNs).
We begin by noting recurrent approaches (e.g. LSTM [9],
GRU [4]) to modelling both visual dialogue and general
dialogue, which have been largely prevalent to date [5, 6, 7,
11, 27]. Typically recurrent neural networks are employed at
two levels – one at the lower level to sequentially generate
the words of a sentence (a question or answer in the case
of dialogue), and one at a higher level to sequence these
sentences together into a dialogue.
Recently however, there has been considerable interest in
convolutional models of language [10, 13, 21]. These models
have shown that they can perform at least as well as recurrent
models, if not better, on a number of different tasks. They
are also much more computationally efficient, and typically

suffer less from issues relating to exploding or vanishing
gradients for which recurrent networks are known [19].
In modelling sentences with convolutions, the tokens
(words) of the sentence are transformed into a stack of
fixed-dimensional embeddings (e.g. using word2vec [17] or
Glove [20], or learnt for a specific task). For a given sentence,
say question qt , this results in an embedding q˚t ∈ RE×L
for embedding size E and sentence length L, where L can
be bounded by the maximum sentence length in the corpus,
with padding tokens employed where required. This twodimensional stack is essentially a single-channel ‘image’ on
which convolutions can be applied in the standard manner in
order to extract an embedding for the entire sentence. Here,
typically convolutions are applied along the sequence dimension (as opposed to the embedding dimension). Note this
similarly applies to the answer at and caption c, with åt
taken to be the answer ‘image’, and c̊ the caption ‘image’.
We extend this idea of viewing sentences as ‘images’
to whole dialogues, which can be represented as a multichannel language embedding. Here, the sequence of sentences itself can be seen as a stack of (a stack of) word
embeddings d̊ ∈ RE×L×2T , where now, the number of
channels in the embedding ‘image’ accounts for the number
of questions and answers in the dialogue. We refer to this
process as “colouring” visual dialogue, by analogy to the
most common meaning given to the image channels – colour.
Our primary motivation for adopting a convolutional approach in modelling dialogue is to explore how it fares when
extended from simpler language tasks [10, 13] to full visual
dialogue. We hence instantiate the following convolutional
models for 1 VD and 2 VD:
Answer [1 VD]: We employ the CVAE formulation from
Eq. (4) and Fig. 3 to iteratively generate answers, conditioned on the image, caption and current dialogue history.
Block [1 VD, 2 VD]: Here, we use the CVAE formulation
from Eq. (3) and Fig. 2 to generate entire blocks of dialogue directly (i.e. dialogue context h = ∅). We allow the
convolutional model to supply the context to the recognition and generative models implicitly instead. We consider
this 2 VD, although this block architecture is also amenable
to iterative generations, and hence can be evaluated on
1 VD as well (see Section 4.4).
Block Auto-Regressive [1 VD, 2 VD]: In this case, we introduce an auto-regressive component to our generative
model in the same sense as recent auto-regressive generative models for images [8, 25]. We augment the Block
model by feeding its output through an auto-regressive
(AR) module which explicitly enforces sequentiality in
the generation of the dialogue blocks. This effectively
factorises the likelihood in Eq. (3) as pθ (d | z, i, c, h) =
 QN

n
1:n−1
pθ d1 | z, i, c, h
where N is the
n=2 pθ d | d
number of AR layers, and d1 is the (intermediate) output
from the standard Block model. Note, again h = ∅, and

Table 1: Data (x) and condition (y) variables for models
A and B/BAR for 1 VD and 2 VD. Models B/BAR can be
evaluated as a block or iteratively (see rightmost column and
Section 4.4). Iterative evaluation can accept the ground-truth
(q/a) or predicted (q̂/â) dialogue history (see Table 2).
Task

Model

Train

Evaluate

Eval method

x

y

x

y

1 VD

A
B, BAR

at
d

i, c, h+t
i, c

∅
{d–qa, d–q â}

i, c, h+t
i, c

−
iterative

2 VD

B, BAR

d

i, c

∅
d–q̂ â

i, c

block
iterative

4.1. Datasets
We use the VisDial [5] dataset (v0.9) which contains Microsoft COCO (COCO) images each paired with a caption
(1 of the 5 COCO captions) and a dialogue of 10 questionanswer pairs. The train/test split is 82, 783/40, 504 images,
respectively. The dialogues are collected by pairs of annotators on Amazon Mechanical Turk (AMT). One “blind"
annotator (provided only the image caption, not the image
itself) acts as the questioner and the other “oracle” annotator
(provided both image and caption) acts as the answerer.

4.2. Baseline
dn refers to an entire dialogue at the n-th AR layer (rather
than the t-th dialogue exchange as is denoted by dt ).

4. Experiments
We present an extensive experimental analysis, both quantitative and qualitative, of our models’ performance in both
visual-dialogue variants: 1 VD, which requires answering a
sequence of image-contextualised questions, and full 2 VD,
where both questions and answers must be generated given
a specific visual context. Our proposed generative models
are denoted as follows:
A – answer architecture for 1 VD
B – block dialogue architecture for 1 VD & 2 VD
BAR – auto-regressive extension of B for 1 VD & 2 VD
A is a generative convolutional extension of our baseline [5]
and is used to validate our methods against a standard benchmark in the 1 VD task. B and BAR , like A, are generative, but
are extensions capable of doing full dialogue generation, a
task much harder than standard visual question-answering.
Importantly, B and BAR are flexible in that despite being
trained to generate a block of questions and answers (h = ∅),
they can be evaluated iteratively for both 1 VD and 2 VD
(see Section 4.4). For clarity, the data and conditioning
variables for each model in training and evaluation are summarised in Table 1. In order to evaluate performance on both
tasks, we propose novel evaluation metrics which augment
those of our baseline [5]. To the best of our knowledge, we
are the first to report models that can generate both questions
and answers given an image and caption, a necessary step
toward a truly conversational agent. Our key results can be
summarised as follows:
• We set state-of-the-art results in the 1 VD task with a high
margin: on the VisDial dataset we obtain a mean rank of
4.85 compared to the current state-of-the-art of 17.06 [5].
• Our block models are able to generate both questions and
answers, a more difficult but more realistic task (2 VD).
• Since our models are generative, we are able to show
highly diverse and plausible question and answer generations based on the provided visual context.

Das et al. [5] introduce the VisDial dataset along with
three models for 1 VD. Their best model, MN-QIH-G, is a
recurrent encoder-decoder architecture which encodes the
image i, the current question qt and the attention-weighted
ground truth dialogue history d1:t−1 . They then sample the
output conditional likelihood distribution to (token-wise) predict an answer at time t. As mentioned earlier, our A model
is a generative and convolutional extension of MN-QIH-G
which we evaluate using the existing ranking-based metrics
of [5]. We can also (iteratively) evaluate our B and BAR
models for 1 VD as detailed in Section 4.4 (see Table 3).

4.3. Network architectures and training
Following the CVAE framework (Section 3) and its convolutional interpretation (Section 3.1), we build all of
our models (A, B and BAR ) with three core components:
an encoder network, a prior network and a decoder network. Figure 4 (top) shows the encoder and prior networks,
and Fig. 4 (middle, bottom) show the standard and autoregressive decoder networks. Below we provide brief details
of each component. A more detailed treatment is included
in the supplementary material.
Prior network
The prior neural network, parametrised
by θ, takes as input the image i, the caption c and the dialogue context. Referring to Table 1, for model A, recall
y = {i, c, h+t } where the context h+t is the dialogue history
up to t-1 and the current question qt . For models B/BAR ,
y = {i, c} (h = ∅, as dialogue context is implicit rather
than explicit). To obtain the image representation, we pass
i through VGG-16 [23] and extract the penultimate (4096dimensional) feature vector. For the caption, we pass c
through a pre-trained word2vec [17] module (we do not
learn these word embeddings). If h 6= ∅, we pass the onehot encoding for each word through a learnable word embedding module and stack these embeddings as described in
Section 3.1. We encode these conditioning variables convolutionally to obtain y (the encoded condition). y is then passed
through a convolutional block to obtain µp and log σp2 , the
parameters of the conditional prior pθ (z | y).

d

Is this image in color?

Yes

How old does the girl appear to be?

Four

Does the donut have sprinkles?

No

Is this taking place inside of a coffee
donut shop?

No

Are there any other people besides the
girl pictured in this image?

No

Is the girl making a mess with the
donut?

Yes

What color hair does the girl have?

Blonde

Has she eaten a lot of the donut?

Yes

Is she smiling?

No

What color are the walls?

White

c

A girl in pink shirt eating a
chocolate frosted donut

µq

Wemb

CNN

w2v

CNN

CNNq

log σ q2

i

VGG
16

h
~~~

Wemb

~~~

µp
CNNy

log σ

z! = µ + εσ

!
d

CNNz

CNNg

AR-N

ε

log σ

2

z! = µ + εσ

WTemb

Is this image in color?

Yes

How old does the girl appear to be?

Four

Does the donut have sprinkles?

No

Is this taking place inside of a coffee
donut shop?

No

Are there any other people besides the
girl pictured in this image?

No

Is the girl making a mess with the
donut?

Yes

What color hair does the girl have?

Blonde

Has she eaten a lot of the donut?

Yes

Is she smiling?

No

What color are the walls?

White

y

N (0,1)

µ

log σ p2

CNN

ε

2

CNNp

y

N (0,1)

µ

y

CNNz

CNNg

!
d

WTemb

Is this image in color?

Yes

How old does the girl appear to be?

Four

Does the donut have sprinkles?

No

Is this taking place inside of a coffee
donut shop?

No

Are there any other people besides the
girl pictured in this image?

No

Is the girl making a mess with the
donut?

Yes

What color hair does the girl have?

Blonde

Has she eaten a lot of the donut?

Yes

Is she smiling?

No

What color are the walls?

White

Figure 4: Convolutional (top) conditional encoder and prior
architecture, (middle) conditional decoder, and (bottom)
auto-regressive conditional decoder architectures, applying
to both one- and two-way visual dialogue (1 VD and 2 VD).
Encoder network
The encoder network, parametrised
by φ, takes x and the encoded condition y (obtained from
the prior network) as input. For model A, x = at while
T
for B/BAR , x = d = h(qt , at )it=1 . In all models, x is
passed through a learnable word embedding module, transforming it into a single-channel answer ‘image’ for A, or a
multi-channel image of alternating questions and answers
for B/BAR . The embedded output is passed through a convolutional block, concatenated with y and forwarded through
a final convolutional block to obtain µq and log σq2 , the parameters of the conditional latent posterior qφ (z | x, y).
Decoder network
The decoder network (for simplicity, the parameters of the prior and decoder network are
subsumed into θ) takes as input a sample z̃ and the encoded condition y. At training, z̃ comes from the Gaussian
qφ (z | x, y) while at test time it comes from pθ (z | y). We
differentiate this as the reconstruction versus the generation
pipeline, accordingly (see supplementary material). The
sample is passed through a transpose-convolutional module, concatenated with y and forwarded through a second
transpose-convolutional module to obtain an intermediate
output volume of dimension E×L×M , where E is the word
embedding dimension, L is the maximum sentence length
and M is the number of dialogue entries in x (M = 1 for A,

M = 2T for B variants). Following this, our models diverge
in their architectures: A and B employ a standard linear layer
which projects the E dimension of the intermediate output
to the vocabulary size V (Fig. 4 (middle)). BAR employs an
autoregressive module followed by a standard linear layer
(Fig. 4 (bottom)). At train time, the V -dimensional network
output is softmaxed and the CE term of the ELBO computed.
At test time, the argmax of the (softmax-ed) output is taken
to be the index of the word token predicted. We share the
weights of the encoder and prior’s learnable word embedding
module with the decoder’s final linear layer.
Autoregressive module
The autoregressive module of
BAR ’s decoder is inspired by PixelCNN [26] which sequentially predicts the pixels in an image along the two spatial
dimensions. In the same fashion as [8], we apply N sizepreserving autoregressive layers to the intermediate volume
of size E × L × 2T , following which a final linear layer
projects E to the vocabulary size V . Each layer employs
masked convolutions on the previous layer’s output feature
volume such that only ‘past’ word embeddings are visible. In
this manner, each layer sequentially predicts 2T ∗ L embeddings of size E, enforcing sequentiality at both the sentenceand dialogue-level. We report numbers for N = {8, 10}.
KL annealing
Motivated by the work of [3] in learning a
continuous latent embedding space for language, we employ
KL annealing in the loss objectives of Eq. (3) (for A) and
Eq. (4) (for B/BAR ). We weight the KL term by α, where α
is linearly interpolated between (0; 1] over 100 epochs. We
further train the models for another 50 epochs with the true
CVAE objectives (i.e. α = 1) .

Network and training hyper-parameters
In embedding sentences as single-channel ‘images’, we use a maximum sequence length of L = 64 and a word embedding dimension of E = 256 (for word2vec embeddings, E = 300).
After pre-processing and filtering the vocabulary, V = 9710
(see supplementary material for further details). We use
the Adam optimiser [15] with default parameters, a latent
dimensionality of 512 and employ batch normalisation with
momentum= 0.001 and learnable parameters. For model A
we use a batch size of 200, and for B and BAR , a batch size
of 40 due to GPU memory availability. We implement our
pipeline using P Y T ORCH [22].

4.4. Evaluation methods for block models
The block models B and BAR are trained to generate
whole blocks of dialogue directly (h = ∅), however, they are
flexible in that they can be evaluated iteratively, lending them
to both 1 VD and 2 VD (see rightmost column of Table 1). A
complete description of the different pipelines employed is
given in the supplementary material.
• Block evaluation [2 VD]. The generation pipeline generates whole blocks of dialogue directly, conditioned on the

Table 2: Iterative evaluation of B/BAR for 1 VD and 2 VD.
Under each condition, the input dialogue block is filled with
ground-truth or predicted history (q/a or q̂/â, respectively),
while future entries are filled with the PAD token.
1 VD
<t
=t
>t

2 VD

d–qa

d–q â

d–q̂ â

(q, a)
(q, PAD)
(PAD, PAD)

(q, â)
(q, PAD)
(PAD, PAD)

(q̂, â)

(PAD, PAD) (q̂, PAD)
(PAD, PAD)

image and caption (i.e. x = ∅ and y = {i, c} for B/BAR
evaluation in Table 1). This is 2 VD since the model must
generate a coherent block of both questions and answers.
• Iterative evaluation. The reconstruction pipeline generates dialogue items iteratively. For time t, we fill the
input dialogue block with the PAD token and slot in the
ground-truth/predicted dialogue history to < t (see below
and Table 2). This future-padded block is encoded with
the image and caption conditions, and then reconstructed
(as done during training). The t-th dialogue item is then
extracted (whether an answer if 1 VD or a question/answer
if 2 VD), and this is repeated T times (or 2T for 2 VD).
Variations of this are as follows:
– d–qa [1 VD]. For time t, the input dialogue block is
filled with the history of ground-truth questions and
answers from 1 to t − 1, along with the current groundtruth question. The current answer slot and all future
dialogue entries are padded. This is equivalent to [5] in
which the ground-truth dialogue history is utilised.
– d–q â [1 VD]. Similar to d–qa, however, for time t,
the input block is filled with the history of ground-truth
questions and predicted answers along with the current
ground-truth question. This is a more realistic 1 VD.
– d–q̂ â [2 VD]. The most challenging and realistic condition in which the input block is filled with the history
of predicted questions and predicted answers. This is
an iterative form 2 VD, thus depending on the iteration,
either a question or an answer will be generated.

4.5. Evaluation and Analysis
Here we present the experimental evaluation of our A,
B, and BAR models on the one-way and two-way visual dialogue tasks (1 VD and 2 VD, respectively). Under 1 VD, the
task is to predict an answer with each time step, given an image, caption and the current dialogue history (Section 4.5.1
and Table 3), while under 2 VD, the task is to predict both
questions and answers given an image and caption (Section 4.5.2 and Table 4). All three of our models are able to
perform the first task , while only B/BAR are capable of the
second task. The following details our evaluation criteria and
analyses our results against current state-of-the-art methods.

4.5.1

One-Way Visual Dialogue (1 VD) task

We evaluate the performance of A and B/BAR on 1 VD using
the candidate ranking metric of [5] as well as an extension of
this which assesses the generated answer quality (Table 3).
Fig. 1 and Fig. 5 show our qualitative results for 1 VD.
Candidate ranking by model log-likelihood [SM ]
The VisDial dataset [5] provides a set of 100 candidate answers {act }100
c=1 for each question-answer pair at time t per
image. The set includes the ground-truth answer at as well
as similar, popular, and random answers. Das et al. [5] rank
these candidates using the log-likelihood value of each under
their model (conditioned on the image, caption and dialogue
history, including the current question), and then observe the
position of the ground-truth answer (closer to 1 is better).
This position is averaged over the dataset to obtain the Mean
Rank (MR). In addition, the Mean Reciprocal Rank (MRR;
1/MR) and recall rates at k = {1, 5, 10} are computed.
To fairly compare against their baseline, we employ our
model A to rank the 100 answer candidates: conditioning
on i, c and h+
t (which includes the dialogue history and
current question), we pass each candidate through our encoder network, sample z̃ ∼ N (z; µq , σq2 ) and reconstruct
it. We score each candidate with its resultant ELBO value
– see Eq. (4) – and rank the candidates accordingly2 . With
this, we achieve a MR of 4.85, a 12.21 point improvement
on the current state-of-the-art [5] (see A (ELBO) versus
MN-QIH-G under SM in Table 3).
The fundamental distinction between our model-based
candidate scoring function and that of Das et al. [5] is that by
employing the ELBO, we compute a lower-bounded estimate
of the marginal conditional log-likelihood of a candidate
answer, as opposed to just its conditional log-likelihood
under the model, as in [5]. Estimating the marginal likelihood involves computing the likelihood, but also crucially
an estimate of the posterior (recognition distribution q). As
mentioned earlier in Section 3, in the case of [5], their posterior, and indeed their prior, is instead simply a Dirac-Delta
(i.e. only a single encoding is evaluated with no stochasticity
coming from the conditioning variable).
Finally, given that the ELBO can be decomposed into a
reconstruction (CE) and regularisation term (KL divergence),
we can additionally probe the effects of each on the ranking.
By using just the CE of the ELBO, our model A still beats
the baseline, but only by around 1 MR point (in log scale),
indicating that both terms are essential to a good estimator
(see A (CE) in Table 3).
Candidate ranking by word2vec cosine distance [Sw2v ]
The evaluation protocol of [5] scores and ranks a given set of
candidate answers, without being a function of the actual an2 Note for S , since the block architectures’ reconstruction pipeline
M
requires a full dialogue block as input (rather than a single answer candidate
as with A), we cannot score and rank candidates under B/BAR .

Table 3: 1 VD evaluation of A and B/BAR on VisDial (v0.9)
test set. Results show ranking of answer candidates based
on the score functions SM and Sw2v .

Question
How old is the
girl?
What race is
the girl?
Is she outside?

Score
function
SM

Sw2v

d–qa

d–q â

Method

MR

MRR

R@1

R@5

R@10

RL-QAbot [6]
MN-QIH-G [5]
A (CE)
A (ELBO)

21.13
17.06
16.78
4.85

0.4370
0.5259
0.3992
0.6229

42.29
30.18
49.86

53.67
62.85
48.94
76.78

60.48
68.88
59.11
86.10

MN-QIH-G [5]
A (RECON)
A (GEN)

31.31
15.36
25.65

0.2215
0.4952
0.3227

16.01
41.77
25.88

22.42
54.67
33.43

34.76
66.90
47.75

B
BAR 8
BAR 10

28.45
25.87
26.30

0.2927
0.3553
0.3422

23.50
29.40
28.00

29.11
36.79
35.34

42.29
51.19
50.54

B
BAR 8
BAR 10

30.57
29.10
29.15

0.2188
0.2864
0.2869

16.06
22.52
22.68

20.88
29.01
28.97

35.37
48.43
46.98

Is her hair
long or short?
What color is
her hair?
Is her hair
curly or
straight?
What is she
wearing?

Z1

Z2

Is the teddy
bear in her
lap?
What color is
the teddy bear?
Is it nice
outside?

Question
How old does
she look?

Z3

Maybe three

Looks
about six

I can’t
tell

Yes

White

Yes

White

Caucasian

Yes

No

Yes

Yes

Short

Short

Short

Short

Blonde

Blonde

Brown

Brown

It’s
straight

Straight

Straight

Straight

Pink shirt,
T-shirt
white pants and jacket
and pants
Yes

No

White
Yes sunny

Groundtruth
answer

Shirt and Jeans like
pants
a pajamas
Yes

Yes

Brown

Tan

Tan and
white

Yes

It looks
lovely

Yes

Z1

Z2

Around
I cannot
seven or tell about
eight
her

She is
about
teenager
around
Yes

No

No

Yes

Any buildings?

No

Yes

No

No

Day

It looks
like it in
image is
in so

Daytime

Day

Is it raining?
What color
umbrella?
Is it open or
closed?

No

Yes

Yes

No

Pink and
clear

Dark
colored
color

White

White

Open

Open

Yes

Open

I can’t
tell

Yes

No

Yes

What color is
her hair?

Dark brown

Brown

Brown

Brown

Is it long or
short?

Long

Short

Short

I’d say
long

Is it sunny?

A young girl swinging with her
teddy bear

Z3

Looks
about six

Any other
people?
Is it day or
night?

swer predicted by the model, ât . This results in the rank of
the ground-truth answer candidate reflecting its score under
the model relative to the rest of the candidates’ scores, rather
than capturing the quality of the answer output by the model,
which is left unobserved. To remedy this, we instead score
each candidate by the cosine distance between the word2vec
embedding of the predicted answer ât and that candidate’s
word2vec embedding. We take the embedding of a sentence
to be the average embedding over word tokens following
Arora et al. [2]. In addition to accounting for the predicted
answer, this method also allows semantic similarities to be
captured such that if the predicted answer is similar (in meaning and/or words generated) to the ground-truth candidate
answer, then the cosine distance will be small, and hence the
ground-truth candidate’s rank closer to 1.
We report these numbers for A, iteratively-evaluated
B/BAR , and also our baseline model MN-QIH-G [5], which
we re-evaluate using the word2vec cosine distance ranking
(see Table 3). In the case of A (GEN), we evaluate answer
generations from A whereby we condition on i, c and h+
t
via the prior network, sample z̃ ∼ N (z; µp , σp2 ) and generate an answer via the decoder network. Here we show an
improvement of 8.68 points in MR over the baseline. On the
other hand, A (RECON) evaluates answer reconstructions in
which z̃ is sampled from N (z; µq , σq2 ) (where ground-truth
answer at is provided). We include A (RECON) merely as
an “oracle” autoencoder, observing its good ranking performance, but do not explicitly compare against it.
We note also, that the ranking scores of the block models
are generally worse (by 3-4 MR points) than those of A.
This is expected since A is explicitly trained for 1 VD which
is not the case for B/BAR . Despite this, the performance
gap between A (GEN) and B/BAR (with d–qa) is not large,
bolstering our iterative evaluation method for the block architectures. Note finally that the B/BAR models perform better
under d–qa than under d–q â (by 2-3 MR points). This

Groundtruth
answer

A young girl holding an
umbrella on the sidewalk

Figure 5: Diverse answer generations from different samples z of the A conditional prior (samples shown by column).
Each generated answer is conditioned on an image from the
VisDial v0.9 test set, along with its caption, question and
dialogue history. Further examples are included in the supplementary material.
is also expected since the task of answering is easier with
access to the ground-truth dialogue history rather than when
only the previously predicted answers (and ground-truth
questions) are provided.
4.5.2

Two-way Visual Dialogue (2 VD) task

The flexibility of our CVAE formulation for visual dialogue
allows us to move from the task of 1 VD to the generation
of both questions and answers (2 VD), either as a block, or
iteratively in sequence. Despite this being inherently more
challenging, our B/BAR models perform well and are able to
generate diverse sets of questions and answers contextualised
by the given image and caption. We show snippets of our
two-way dialogue generations in Fig. 6.
In evaluating our models for 2 VD, the candidate ranking
protocol of [5] which relies on a given question to rank the
answer candidates, is no longer usable when the questions
themselves are being generated. This is the case for B/BAR
block evaluation, which has no access to the ground-truth
dialogue history, and the d–q̂ â iterative evaluation, when the
full predicted history of questions and answers is provided
(Table 2). We therefore look directly to the CE and KL terms

Are it sunny?

Yes

Can you see the sky?

No

Any birds?

No

Can you see any people?

No

Is this a sunny photo?

Yes

Is the sheep in?

Yes
Its is
brown

What color is the sheep?
Any people?

No

Is it sunny?

Yes

What is the the doing?

It is

Is the sheep made?

Yes, is

Sheep standing near orange netting in grassy field
Are there people?

Yes

Is it a professional photo?

Yes

Color color is the uniform?

White

Can you see the team?

No

Are there any people?

No

Can there?

No

Can you see the bases?

No

How many people are you see

Two

Can you see the sky?

Yes

Is the sky visible?

Yes

Is the people visible?

Yes

Does he have a?

Yes is is

Is it a?

Yes is is

A baseball player for the Chicago Cubs stands at home plate

What color is the cabinets?

White

Are there any people in the table?

No

Does the window have curtains

Yes

Is the fridge on?

Yes

Are there any people in the?
Is the magnets on?
What color is the walls?
How many chairs are there?
Is there any people?
Is it daytime?

No
No
White
Two
No
Yes

Is there?

No

How pics the?

No

An image of a kitchen loft style setting

What color are the uniforms?
Can you see any sky?

One is white
No

Is it people visible?

Yes

Is this a game?

Yes

Is the photo in color?
Is the photo close?
Can you see the bases?
Can you see the ball?
Is the person wearing a hat?

No
Yes, it
No
No
Yes

Can you see the the

No

Can you see the people?

No

Two guys playing baseball, with trees in
the back
Is it outside?

Yes

What kind of dog is it?

Can’t

What color is dog?

White

What color is the dog?

White

Any people?

No

Is it sunny?

No

Can you see the sky?

No

Can you see the sky?

No

Are there any people?
Is the water calm?

What color is the car?
Are there a?

No
Yes

Black
No

A dog sitting on the inside of a white boat

Figure 6: Qualitative examples of two-way dialogue generations from the B/BAR models. Strings (without break) of
coherent questions and answers are shown in a single colour.
Different colours indicate different z̃ samples from which
the dialogues were generated. Failure samples are shown
in white in the bottom two rows of each image. Further
examples are included in the supplementary material.

of the ELBO to compare our methods in the 2 VD task, as well
as propose two new metrics, simc,q and sim , to assess the
quality of the generated dialogues:
• Question relevance (simc,q ). We expect a generated
question to query an aspect of the image, and we use the
presence of semantically similar words in both the question and image caption as a proxy of this. We compute the
cosine distance between the (average) word2vec embedding of each predicted question qt and that of the caption
c, and average over all T questions in the dialogue. A
value closer to 1 indicates a semantic similarity between
the questions and the image caption.
• Latent dialogue dispersion (sim ). For a generated
dialogue block dg , sim computes the KL divergence
DKL (qφ (z|dg , i, c) k qφ (z|d, i, c)), measuring how close
the generated dialogue is to the true dialogue d in the
latent space, given the same image i and caption c.
Looking into Table 4, which also shows B/BAR trained
with permutations of the image i and caption c (denoted
by + if included in the condition, and - otherwise), interesting trends emerge. Firstly, we note the decrease in CE and
KL as conditions (i, c) are excluded from the model. This
is expected since the task of dialogue generation is made
simpler without the constrains of an explicit visual/textual
condition. Secondly, we observe a decrease in the loss terms
as the auto-regressive capacity of the model increases (none
→ 8 → 10), suggesting that explicitly enforcing sequentiality in the dialogue generations is useful. Looking at sim ,
within a particular model, the dispersion values are typically
larger for the harder task (i.e. when no dialogue context is
provided). We also observe that as the number of AR layers
is increased, dispersion also increases, suggesting AR improves the diversity of the model outputs, and avoids simply
recovering data close to that observed at train time.
While the proposed metrics provide a novel means to evaluate dialogue in a generative framework, like all languagebased metrics, they are not complete. The question relevance
metric, simc,q , may stagnate over rounds of dialogue, and
neither metric precludes redundant or nonsensical questions.
Instead, we intend for these metrics to augment the growing
bank of metrics available to evaluate dialogue and language
models. Further evaluation protocols including i) using auxiliary tasks, similar in spirit to the image-retrieval task of [6],
to drive and evaluate the dialogues, and ii) turning to human
evaluators on AMT to rate the generated dialogues, will also
be interesting to explore in the future in order to paint a more
complete picture of our models.

5. Conclusion
In this work we propose F LIP D IAL, a generative convolutional model for visual dialogue which is able to generate
answers (1 VD) as well as generate both questions and answers (2 VD) based on a visual context. In the 1 VD task,

Table 4: 2 VD evaluation on VisDial (v0.9) test set for B/BAR
models. Note that d left blank indicates the block evaluation method, when a whole dialogue is generated given only
an image and its caption, while d–q̂ â indicates the iterative evaluation method when previously generated questions
and answers are included in the dialogue history (see Section 4.4). The + and - indicate models trained with and
without respective conditions, image i and caption c.
Method

B

BAR 8

BAR 10

i

c

d

CE

KLD

simc,q

sim

+

+

−
d–q̂ â

31.18
25.40

4.34
4.01

0.4931
0.4091

14.20
1.86

-

+

−
d–q̂ â

29.09
24.59

3.26
3.05

0.4889
0.3877

11.23
3.45

+

-

−
d–q̂ â

28.60
29.85

4.26
4.66

0.4634
0.4221

15.56
3.54

-

-

−
d–q̂ â

19.92
19.34

6.42
0.00

0.4590
0.4638

6.34
0.00

+

+

−
d–q̂ â

28.81
26.60

2.54
2.29

0.4878
0.3884

31.50
2.39

-

+

−
d–q̂ â

30.59
26.15

2.72
2.77

0.4889
0.3758

43.17
3.57

+

-

−
d–q̂ â

31.51
21.41

2.91
2.68

0.4602
0.4453

24.75
5.49

-

-

−
d–q̂ â

20.32
21.53

2.77
2.99

0.4464
0.4419

0.26
0.10

+

+

−
d–q̂ â

28.49
24.93

1.89
1.80

0.4927
0.4101

44.34
2.35

-

+

−
d–q̂ â

30.83
28.59

2.53
2.52

0.4951
0.3903

38.60
1.91

+

-

−
d–q̂ â

30.18
28.32

2.89
2.44

0.4592
0.4334

100.81
6.73

-

-

−
d–q̂ â

19.60
19.17

0.00
0.00

0.4585
0.4614

0.00
0.00

we set new state-of-the-art results by a large margin, and
in the 2 VD task, we are the first to establish a baseline, for
which we propose two novel metrics to assess the quality
of the generated dialogues. In addition, we propose and
evaluate our models under a much more realistic setting for
both visual dialogue tasks in which the predicted rather than
ground-truth dialogue history is provided at test time. This
challenging setting is more akin to real-world situations in
which dialogue agents must be able to evolve with their predicted exchanges. We emphasize that research focus must
be directed here in the future. Finally, under all cases, the
sets of questions and answers generated by our models are
qualitatively good: diverse and plausible given the visual
context. Looking forward, we are interested in exploring
additional methods for enforcing diversity in the generated
questions and answers, as well as extending this work to
explore recursive models of reasoning for visual dialogue.

References
[1] S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L.
Zitnick, and D. Parikh. VQA: Visual Question Answering.
In ICCV, 2015. 2
[2] S. Arora, Y. Liang, and T. Ma. A simple but tough-to-beat
baseline for sentence embeddings. In ICLR, 2017. 8
[3] S. R. Bowman, L. Vilnis, O. Vinyals, A. M. Dai, R. Jozefowicz, and S. Bengio. Generating sentences from a continuous
space. arXiv preprint arXiv:1511.06349, 2015. 6
[4] J. Chung, C. Gulcehre, K. Cho, and Y. Bengio. Empirical
evaluation of gated recurrent neural networks on sequence
modeling. arXiv preprint arXiv:1412.3555, 2014. 4
[5] A. Das, S. Kottur, K. Gupta, A. Singh, D. Yadav, J. M. Moura,
D. Parikh, and D. Batra. Visual Dialog. In CVPR, 2017. 2, 3,
4, 5, 7, 8, 11, 13
[6] A. Das, S. Kottur, J. M. Moura, S. Lee, and D. Batra. Learning
cooperative visual dialog agents with deep reinforcement
learning. arXiv preprint arXiv:1703.06585, 2017. 2, 4, 8, 9
[7] A. Das, S. Kottur, J. M. Moura, S. Lee, and D. Batra. Learning
cooperative visual dialog agents with deep reinforcement
learning. In ICCV, 2017. 4
[8] I. Gulrajani, K. Kumar, F. Ahmed, A. A. Taiga, F. Visin,
D. Vazquez, and A. Courville. Pixelvae: A latent variable
model for natural images. arXiv preprint arXiv:1611.05013,
2016. 4, 6, 13
[9] S. Hochreiter and J. Schmidhuber. Long short-term memory.
Neural Computation, 9(8):1735–1780, Nov. 1997. 4
[10] B. Hu, Z. Lu, H. Li, and Q. Chen. Convolutional neural network architectures for matching natural language sentences.
In NIPS, pages 2042–2050, 2014. 2, 4
[11] U. Jain, Z. Zhang, and A. Schwing. Creativity: Generating diverse questions using variational autoencoders. arXiv preprint
arXiv:1704.03493, 2017. 2, 4
[12] J. Johnson, A. Karpathy, and L. Fei-Fei. Densecap: Fully
convolutional localization networks for dense captioning. In
The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016. 2
[13] N. Kalchbrenner, E. Grefenstette, P. Blunsom, D. Kartsaklis,
N. Kalchbrenner, M. Sadrzadeh, N. Kalchbrenner, P. Blunsom, N. Kalchbrenner, and P. Blunsom. A convolutional neural network for modelling sentences. In Proceedings of the
52nd Annual Meeting of the Association for Computational
Linguistics, pages 212–217. Association for Computational
Linguistics, 2014. 2, 4
[14] A. Karpathy and L. Fei-Fei. Deep visual-semantic alignments
for generating image descriptions. In Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition,
pages 3128–3137, 2015. 2
[15] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference
on Learning Representations (ICLR), 2014. 6
[16] D. P. Kingma and M. Welling. Auto-encoding Variational
Bayes. In Proceedings of the 2nd International Conference
on Learning Representations (ICLR), 2014. 2, 13
[17] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean.
Distributed representations of words and phrases and their
compositionality. In NIPS, pages 3111–3119, 2013. 4, 5, 11

[18] N. Mostafazadeh, I. Misra, J. Devlin, M. Mitchell, X. He,
and L. Vanderwende. Generating natural questions about an
image. arXiv preprint arXiv:1603.06059, 2016. 2
[19] R. Pascanu, T. Mikolov, and Y. Bengio. On the difficulty of
training recurrent neural networks. In International Conference on Machine Learning, pages 1310–1318, 2013. 4
[20] J. Pennington, R. Socher, and C. D. Manning. Glove: Global
vectors for word representation. In EMNLP, pages 1532–
1543, 2014. 4
[21] N.-Q. Pham, G. Kruszewski, and G. Boleda. Convolutional
neural network language models. In EMNLP, pages 1153–
1162, 2016. 2, 4
[22] PyTorch. http://pytorch.org/, 2017. Accessed: 201711-4. 6
[23] K. Simonyan and A. Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition. CoRR,
abs/1409.1556, 2014. 5, 11
[24] K. Sohn, H. Lee, and X. Yan. Learning structured output
representation using deep conditional generative models. In
NIPS, pages 3483–3491, 2015. 2, 3
[25] A. van den Oord, N. Kalchbrenner, L. Espeholt, O. Vinyals,
A. Graves, et al. Conditional image generation with pixelcnn
decoders. In Advances in Neural Information Processing
Systems, pages 4790–4798, 2016. 4
[26] A. van den Oord, N. Kalchbrenner, and K. Kavukcuoglu.
Pixel recurrent neural networks. In NIPS, 2016. 6, 13
[27] T. Zhao, R. Zhao, and M. Eskenazi. Learning discourselevel diversity for neural dialog models using conditional
variational autoencoders. arXiv preprint arXiv:1703.10960,
2017. 2, 4

B. Extended Quantitative Results on 1 VD task
Table 3 in the main paper evaluates A and B/BAR in the
task of one-way visual dialogue (1 VD). Here we shed light
on these numbers and the metrics used to obtain them. We
also present a more extensive quantitative analysis of B/BAR
in the 1 VD task (see Table 5).
Evaluating B/BAR on 1 VD
We extend Table 3 withTable 5, which further compares B/BAR under the iterative
evaluation settings of d–qa and d–q â, using the CE and
KL terms of the ELBO and our two new metrics, simc,q and
sim . We observe that B/BAR (d–qa) shows superior performance of around 7-10 points in MR over B/BAR (d–q â),
and also improves in MRR and recall rates. This is expected
since the ground-truth rather than predicted answers are included in the dialogue history (along with the ground-truth
questions). The metrics simcap,q and sim , on the other
hand, show very little performance difference across the two
evaluation settings. We also note that ranking performance
is worse when both image i and caption c are excluded from
conditioning variable. This does not, however, correlate
with the CE and KL terms of the loss which are lower for a
condition-less setting. We attribute this to the model being
transformed from a CVAE to a VAE, hence lifting the burden
of capturing the conditional posterior distribution (i.e. the
KL is now between an unconditional qφ (z | x) and N (0, 1)).
Interestingly, however, excluding either the image or the caption achieves similar performance to when both are included,
indicating that the caption acts as a good textual proxy of
the image (a reassurance of our simc,q metric).

A. Glossary
block dialogue/architecture Models B/BAR are built and
trained for the task of two-way visual dialogue (2 VD) with
data x = d and conditioning variable y = {i, c}. Since d
T
refers to the whole dialogue sequence/block h(qt , at )it=1
we refer to B/BAR as block architectures.
generation This represents the scenario when only the conditioning variable y is available at test time. In this case,
the decoder network receives a sample z̃ ∼ pθ (z | y), a
multivariate Gaussian parametrised by µp and exponentiated log σp2 learned using the prior network. We call the
decoded output d̂ a generation.
reconstruction Differing from a generation, both y and
d are available. The decoder network receives a sample
z̃ ∼ qφ (z | d, y), a multivariate Gaussian parametrised
by µq and exponentiated log σq2 learned using the encoder
network. We call the decoded output d̂ a reconstruction.
The reconstruction pipeline is used during training when
the input d and the conditioning variable y are available.
Note, this pipeline is also used when B/BAR are evaluated
iteratively (see Section 4.4).

C. Network architectures and training
The following section provides detailed descriptions of
the architectures of our models A, B and BAR . The descriptions are dense but thorough. We also include further details
of our training procedure. Where not explicitly noted, each
convolutional layer is proceeded by a batch normalisation
layer (with momentum = 0.001 and learnable parameters)
and a ReLU activation.
The prior neural network, parametrised
Prior network
by θ, takes as input the image i, the caption c and the dialogue context. For the model A, this context is h+t , containing the dialogue history up to t-1 and the current question
qt . For models B/BAR , the dialogue context is the null set
(h = ∅). To obtain the image representation, we scale and
centre-cropped each image to 3 × 224 × 224 and feed it
through VGG-16 [23]. The output of the penultimate layer
is extracted and `2 -normalised (as in [5]) to obtain a 4096dimensional image feature vector. For the caption, we pass c
through a pre-trained word2vec [17] model (we do not learn
these word embeddings) to obtain c̊ ∈ R300×L where L is
the maximum sentence length (L = 64). For the dialogue
context (relevant only in the case of A) we pass the one-hot

Table 5: 1 VD evaluation of B/BAR on VisDial (v0.9) test set. Results show ranking of answer candidates based on the Sw2v
scoring function. Note that d–qa indicates the iterative evaluation method when the ground-truth dialogue history is provided,
while d–q â, the iterative evaluation method when the ground-truth question and predicted answer history is provided (see
Section 4.4). The + and - indicate models trained with and without respective conditions, image i and caption c.
Method

B

BAR 8

BAR 10

i

c

+

+

+
+
-

CE

KLD

MR

MRR

R@1

R@5

R@10

simcap,q

d–qa

18.87

4.36

28.45

0.2927

23.50

29.11

42.29

0.4374

2.68

+
+
-

d–q â

25.10
16.80
21.02
19.35

4.02
3.13
4.71
13.34

30.57
27.76
29.82
29.00

0.2188
0.3243
0.2144
0.3026

16.06
26.59
15.25
24.36

20.88
33.21
21.07
30.70

35.37
47.65
34.96
47.62

0.4118
0.4491
0.4551
0.4638

2.42
4.48
5.44
6.17

+

+

d–qa

15.11

2.53

25.87

0.3553

29.40

36.79

51.19

0.4703

4.30

+
+
-

+
+
-

d–q â

25.70
16.19
20.39
20.92

2.21
2.80
2.89
2.84

29.10
26.04
28.99
28.79

0.2864
0.3566
0.3024
0.3045

22.52
29.62
24.33
24.46

29.01
36.75
30.74
30.99

48.43
50.62
47.17
48.10

0.3885
0.4626
0.4461
0.4442

3.47
4.17
8.16
0.18

+

+

d–qa

16.04

1.89

26.30

0.3422

28.00

35.34

50.54

0.4708

4.84

+
+
-

+
+
-

d–q â

24.77
19.97
20.39
19.17

1.81
2.58
2.79
0.00

29.15
26.84
27.27
29.00

0.2869
0.3212
0.3157
0.3026

22.68
25.90
25.45
24.36

28.97
32.92
32.26
30.70

46.98
47.68
47.87
47.62

0.4058
0.4424
0.4707
0.4614

2.85
5.95
13.22
0.00

encoding of each word through a learnable word embedding
module. We stack these embeddings as described in Section 3.1 of the main paper to obtain h˚+t ∈ RE×L×K , where
E is the word embedding dimension (E = 256), L is the
maximum sentence length (L = 64) and K is the number of
dialogue entries at time t. We encode these inputs convolutionally to obtain y (the encoded condition) as follows: c̊ is
passed through a convolutional block (output size 64 × 8 × 8)
and concatenated with the image feature vector (reshaped to
64×8×8). The concatenated output is passed through a convolutional block to obtain the jointly encoded image-caption
(output size 64 × 8 × 8). If h 6= ∅, then the context is passed
through a convolutional block (output size 64 × 8 × 8) and
is concatenated with the encoded image-caption and passed
through yet another convolutional block to get the encoded
image-caption-context (output size 64 × 8 × 8). We call this
the encoded condition y. The encoded condition y is then
passed through a further convolutional block (output size
256 × 4 × 4) followed by two final convolutional layers (in
parallel) to obtain µp and log σp2 , respectively, the parameters of the conditional prior pθ (z | y). At this stage, µp and
log σp2 are both of size 512×1×1 (the latent dimensionality).
At test time, a sample is obtained via z̃ ∼ N (z; µp , σp2 ) and
is passed to the decoder in order to generate a sample ât (for
A) or d̂ (for B/BAR ).
Encoder network
The encoder network, parametrised
by φ, takes x as input along with the encoded condition,
y, obtained from the prior network. For model A, x =

sim

at and y = {i, c, h+t }. For models B/BAR , x = d =
T
h(qt , at )it=1 and y = {i, c}. In all models, x is passed
through a learnable word embedding module, and the word
embeddings stacked (see Section 3.1 in the main paper) to
obtain x̊ ∈ RE×L×M , where E = 256, L = 64 and M is
the number of entries in x (for A, M = 1 and for B/BAR
M = 2T ). In this way, we transform x into a single-channel
answer ‘image’ in the case of A, and a multi-channel image
of alternating questions and answers in the case of B/BAR .
x̊ is then passed through a convolutional block (output size
64 × 8 × 8), the output of which is concatenated with y and
forwarded through another convolutional block (output size
256 × 4 × 4). This output is forwarded through two final
convolutional layers (in parallel) to obtain µq and log σq2 , the
parameters of the conditional latent posterior qφ (z | x, y).
Here µq and log σq2 are both of size 512 × 1 × 1.
At train time, the KL divergence term of the ELBO is
computed using {µq , σq } (from the encoder network) and
{µp , σp } (from the prior network).
Decoder network
The decoder network (for simplicity,
the parameters of the prior and decoder network are subsumed into θ) takes as input a sample z̃ and the encoded
condition y. During training, z̃ is sampled from a Gaussian
parametrised by the µq and exponentiated log σq2 outputs of
the encoder network. This distribution is qφ (z | x, y). At
test time, z̃ is sampled from a Gaussian parametrised by the
µp and exponentiated log σp2 outputs of the prior network.
This distribution is pθ (z | y). At both train and test time, we

employ the commonly-used ‘re-parametrisation trick’ [16] to
compute the latent sample as z̃ = µ + σ where  ∼ N (0, 1)
and µ and σ correspond to those derived from the encoder
or prior network as described above.
The sample z̃ is then transformed through a transposeconvolutional block (output size 64 × 8 × 8), concatenated
with y and forwarded through a convolutional block (output
size 64 × 8 × 8). This output is forwarded through a second
transpose-convolutional block, producing an intermediate
output volume of dimension M × E × L which we permute
to match the size of x̊. As before, E = 256, L = 64 and
M = 1 (for A) or M = 2T (for B/BAR ).
Following this, our models diverge in architecture: A
and B employ a standard linear layer which projects the
E dimension of the intermediate output to the vocabulary
size V . The BAR model instead employs an autoregressive
module (detailed below) followed by this standard linear
layer. At train time, the V -dimensional network output is
softmaxed and used in the computation of the CE term of
the ELBO. At test time, the argmax of the (softmax-ed)
output is taken to be the index of the word token predicted.
We share the weight matrices of the decoder’s final linear
layer and the encoder and prior’s learnable word embedding
module (which are the same size by virtue of our network
architecture) with the motivation that language encoders and
decoders should share common word representations.
Autoregressive block
The autoregressive (AR) block
(AR-N in Fig. 4 - bottom) in BAR ’s decoder is inspired by
PixelCNN [26] which sequentially predicts the pixels in an
image along the two spatial dimensions. In the same fashion,
we use an autoregressive approach to sequentially predict
the next sentence (question or answer) in a dialogue. Since
our framework is convolutional with sentences viewable as
‘images’, our approach can similarly be adapted from that
of [26, 8]. We first reshape the intermediate output of the
decoder to E × L∗M (essentially ‘unravelling’ the dialogue
sequentially into a stack of its word embeddings). We then
apply a size-preserving masked convolution to the reshaped
output (followed by a learnable batch normalisation and a
ReLU activation). We call this triplet an AR layer. The
masked convolution of the AR layer ensures that future rows
(i.e. future E-dimensional word embedding) are hidden in
the prediction of the current row/word embedding. We apply
N AR layers in this way with each layer taking in the output
of the previous AR layer. Following the AR-N block, a
linear layer projects the final output’s E dimension to the
vocabulary size V . We report numbers for N = {8, 10}.
We base our implementation of the AR block on a publiclyavailable implementation of PixelCNN 3 .
3 https://github.com/jzbontar/pixelcnn-pytorch

D. Dialogue preprocessing
The word vocabulary is constructed from the VisDial
v0.9 [5] training dialogues (not including the candidate answers). The dialogues are preprocessed as follows: apostrophes are removed, numbers are converted to their worded
equivalents, and all exchanges are made lower-case and either padded or truncated to a maximum sequence length
(L = 64). The vocabulary is also filtered such that words
with a frequency of <5 are removed and replaced with the
UNK token. After pre-processing and filtering, the vocabulary
size is V = 9710.

E. Extended Qualitative Results
We present additional qualitative results for the A model
in Figs. 7 and 8 (1 VD task) and for the BAR 10 model (under
the block evaluation setting) in Figs. 9 and 10 (2 VD task)
. Note that for both, different colours indicate generations
(ât for A and d̂ for B/BAR ) from different samples of z. In
Figs. 9 and 10, whole generated dialogue blocks are shown
with coloured sections indicating subsets exhibiting coherent
question-answering and white sections indicating subsets
that are not entirely coherent.

Question

Groundtruth
answer

How old is the
girl?

Maybe three

I can’t
tell

Looks
about six

She is
maybe

I can’t
tell

Yes

White

All

Yes

White

White

Caucasian

Yes

Yes

No

Yes

Yes

Yes

Short

Short

Short

Yes

Short

Short

Blonde

Brown

Brown

What race is
the girl?
Is she
outside?
Is her hair
long or short?

Z1

Z2

Z3

Z4

Z5

Blonde

What color is
her hair?

Blonde

Blonde

Is her hair
curly or
straight?

It’s
straight

Straight

Straight

Short

Straight

Straight

What is she
wearing?

Pink shirt,
white pants

Clothes,
shirt and
top

T-shirt
and jacket
and pants

T-shirt
and pants

Shirt and
pants

Jeans like
a pajamas

Yes

There

No

Yes

Yes

Yes

Tan

Tan and
white

It looks
lovely

Yes

Is the teddy
bear in her
lap?
What color is
the teddy
bear?

Brown

White

Is it nice
outside?

Yes sunny

Yes

Groundtruth
answer

Question

Tan

Brown

It looks
sunny

Yes

Z1

Z2

Z3

Z4

A young girl swinging with her
teddy bear

Z5

Around
seven or
eight

I cannot
tell
about her

Looks
about six

She is
about
teenager
around

I can’t
tell

I can’t
tell

Any other
people?

No

No

Yes

Yes

No

I don’t
see anyone

Any buildings?

No

Yes

No

No

Secret
can’t tell

Yes

Day

It looks
like it
in image
is in so

Daytime

Day

Day

Yes

How old does
she look?

Is it day or
night?
Is it raining?
What color
umbrella?

No

Yes

Yes

No

No

No

Pink and
clear

Dark
colored
color

White

White

Grey

White

Open

Yes

Open

Open

Open

It is open

Yes

No

Yes

Yes

No

Brown

Brown

Brown

Brown

Brown

I’d say
long

Short

Yes

Is it open or
closed?

I can’t
tell

Is it sunny?
What color is
her hair?

Dark brown

Is it long or
short?
Question

Long
Groundtruth
answer

Z1

Short

Z2

Short

Z3

Z4

Z5

Are there any
animals
present?

No animals

No

No

Yes

No

Are there any
kids?

Two kids

Yes

No

Yes

Yes

I

Boys

Boy

Boys

I

Boys

Yes

Are there any
adults?

One adult

Yes

Yes

Yes

Yes

Yes

What are they
doing?

Adult is
flying kite
and boys
are
watching

Playing

She is
watching
parasols

Spectating

They are
playing
watching

Yes

Yes

I can’t
tell

Yes

Yes

Boys or girls?

Is it sunny?

I can’t
see their
faces

No

No

Are there
trees?

Yes

Yes

Yes

Yes

Yes

There are
no
background

Is kite
colorful?

Yes

No

Yes

Yes

Yes

Yes

Is kite in
air?

Yes

Yes

No

Yes

No

Are boys
smiling?

Yes

Yes

No

Yes

Yes

Yes
Yes

A young girl holding an
umbrella on the sidewalk

There is a woman playing with a kite with two boys

Figure 7: Examples of diverse answer generations from the A model for the 1 VD task.

Question

Groundtruth
answer

Z1

Z2

Z3

Z4

Z5

Is the man in
a park?

I can’t
tell

It appears
so

Yes

Yes

Yes

He is in

How old is the
man?

Thirtys
maybe

Locate

Maybe late
thirtys

Thirtys

Thirty

Twentys

Has the
frisbee left
his hand?

Yes

Yes

Yes

Yes

Yes

Yes

Can you tell
what he’s
throwing it
towards?

No

No

I can’t
tell, is
ball

I can’t
tell

No

Yes

What does
’casually
thrown’ mean?

It looks
like he
just tossed
it

Casually

I

He has in
a
skateboard
shirt

Casually

What color is
the frisbee?

White

White

Blue

It is
white

White

What is the
man wearing?
Is he wearing
jeans?

T-shirt

T-shirt
and shorts

He looks
casual

Shorts

T-shirt
and shorts
shorts

I can’t see
his pants

No

Shorts

Yes

No

Yes

No

No

No

No

No

Do you see any
animals?

No

Is it a tall
wall?
Is there only
one polar bear?

Groundtruth
answer

No

No

Z1

Z2

Yes, it goes
underwater

Yes

Yes, just
one

Yes

No

Any people
there?
Do you see quite
a lot of water?

White

A white
screen
print tee

Are there
other people
around?

Question

At least
relief
casually

Yes the
polar bear
is
underwater

Z3

No

No

No

Z4

No

Z5

Yes

Yes can’t
tell

No

No

One

Yes

One

Yes

Yes

I can’t
see any

No

No

Yes

Yes

I can’t
see the
ground

No

Yes can’t
see any

Yes can’t
tell

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Can you see the
sky?

No

No

Is this picture
in color?

Yes it is

Yes

Yes

I don’t Looks like
think it is
relaxed

I

Yes

No

Yes

No

Yes

There are
no
buildings

I

No

No

There see
zero

Yes

Yes

No

Yes

Yes

Yes

Is this a
professional
picture?
Do you see any
buildings?

No

Any trees?

No

Is this a closeup type of
photo?

No

Question

Groundtruth
answer

Z1

Z2

How many
children are
there?

Many

There are
six

All boys or
girls?

Both
Six

How old do
they look?
Are they
outside?
Is it sunny
out?
Are they in a
field?
Are there many
adults around?
Is there only
one ball?
Are there
soccer nets?
What’s in the
background?

A man casually throws a frisbee into the air

Z3

It looks
Yes like it is
in frame

Z4

Z5
More than
ten ten

One

Three

There

I can’t
tell

Boy

Girl and
each

It appears
boy girl

They are
so

Five

Ten-twelve

Twenty

Yes

Five looks
six

Yes

Yes

Yes

Yes

Yes

Yes

Yes
No

Yes

I

Maybe

Yes

Yes

Yes

No

No

Yes

Zero

No

Yes

No

Yes

No

Yes

Yes

Yes

Yes

Yes

Yes

No

Yes

I can’t
say

Yes

I see the
grass

There are
numerous
snow trees

Lots

I see some
grass

Not visible
Fence

No

Trees

Yes

A polar bear swimming in water near a rock wall

Small children in red and blue uniforms, kicking a red soccer ball

Figure 8: Examples of diverse answer generations from the A model for the 1 VD task – continued.

Sheep standing near orange netting in grassy field

How many sheep are
there?

Four

Is this a color?

Yes

Can you see any
people?

No

Are you any any water?

No

Can you see any?

No

Is the photo in
color?

Yes

What color are they?

Brown

Can you see trees?

No

What is the fence
made?

I is

Any people?

No

Any people?

No

Can you see any?

No

What are the the?

Grazing

Is it sunny?

No

How many sheep are
there?

I are
It

Are there people?

No

Are it sunny?

Yes

What color is the
shacks?

Is it sunny?

Yes

Any people?

No

Does the grass have?

Yes

Can any animals?

No

Any people?

No

Can you see any sky?

No

Is the grass green?

Yes

Do you see any sky?

No

Are any any?

No

What time the day?

Can’t
tell

Is any sunny?

Yes

Are they eating?

Yes

Can you see the ball?

Yes, I

Is this a
professional game?

Any many players are? One

Is the people male?

Yes

Is the player a?

Can you see any sky?

No

Is it a a a?

Yes, is
is

Is the a?

Yes

What color is the
uniforms?

White

Can you see the
baseball?

Yes, is
is

Can you see the
pitcher?

No

Is there grass in?

Yes

What is the the
wearing?

It is
wearing

Is the a in?

Yes

How many people are
there?

Two

How is the the bat?

I is is

Is it a in the?

Yes

Yes

Is there any other
people?

Yes

What color is the
batters?

White

Yes

Can you see the
pitcher?

No, the
see

Is there people in
the?

No

What color is the sky?

The

Can you see the
scoreboard?

No

Is the pitcher in?

Yes

Is it sunny?

Yes

Can you see the sky?

No

Is it sunny?

Yes

What color is?

White

What color is the
fridge?

White

Can you see the?

No

Is there a fridge?

Yes

What color is?

White

What appliances?

Microwave

Is there any people?

No

Can you see the sink?

Yes

Is the fridge on?

Yes

Is it a?

Yes

Is there a window?

No

Is it a?

Yes

No

Any people?

No

Is the fridge have
magnets?

Yes
Colgate

Can you see any?

Is there a window?

An image of a kitchen loft style setting

Is there a?

No

Is there?

No

Can you see what
time?

Is there a?

Yes

How pics the?

No

Is there a on the?

No

Is the fridge on?

Yes

What color is the
walls?

White

Can you tell what
time?

No

Can you see the window?

No

Any windows?

No

Is there a?

Yes

Is it a?

Yes

How about on?

Two

Are there any people? No

Is the photo in color?

Yes

What color are the
uniforms?

One is
white

What is the the of?

One

Is it a professional
photo?

Yes

Are they wearing?

Yes

Are there see other

Yes

Yes

Are they people
uniforms?

Yes

Is there a?

Yes

Is they wearing
uniforms?

Yes

Is there trees?

Yes

What color are the
uniforms?

White

Is it sunny?

Yes

Does the have wearing
Yes
hair

Are the men wearing or?
Is any of wearing?
Are there any people in
the?

Two guys playing baseball,
with trees in the back

Yes

Is the person wearing?

Yes, is
is

Is it sunny sunny?

A baseball player for the Chicago Cubs stands at home plate

Yes

Yes
Yes
Yes

Are there a?

How many men?

Two

Are the men wearing
wearing uniforms?

Yes

Are the wearing
wearing?

Yes
they
does

Are there see other
people?

No

Are they wearing
cleats?

Yes

Can you see any ball? No

What color are the
uniforms?

White

Are there a fence?

Yes

Can you see any
coaches?

Do the have the the?

Yes

Can you see the?

No

Do you see the color? Yes

What color the
wearing?

A

No

Figure 9: Diverse two-way dialogue generations from the BAR 10 model (block evaluation) for the 2 VD task.

Man and a boy playing
ball in the grass

Is the picture in
color?

Yes

How many people are
there?

Two

What color is the
man’s?

White

Is the photo in?

Yes

What are they?

I

What is the man man?

I

What color is the
frisbee?

White

Can you see the
frisbee?

Yes

How old is the man?

I

Can there?

No

What color of the
frisbee?

Blue

Can you see any ball? Yes

Is the grass visible?

Yes

Is there people?

Yes

Is there any other
people?

No

Is the sunny?

Yes

Is it sunny?

Yes

Does the man have a?

No

Is it sunny?

Yes

What color is the?

I

Is the man wearing a
hat?

No

Is there other people?

No

Is there sunny?

Yes

Does he man have a?

No

What color is the
frisbee?

White

How many people are
there?

I

Are there see any
other people?

No

Is the man green?

Yes

Is they having fun?

Yes

Can you see any sky?

No

A dog sitting on the inside of a white boat

It looks
like a mutt

Does he have a
collar?

Yes

Can you see what
breed?

No

Is it people boat
the the boat?

No

Is there oars?

No

How the boat the
the?

I

I is

Is it sunny?

Yes

Is it a or or?

It Is

How many barges?

No

Yes

What is of day is?

It is

Any people?

No

No

What color is the boat?

It

Any?

No

Yes

Are there only one?

Yes

How is of the is?

I

What kind of dog is it?

I is
like

Can you see the breed?

No

Does it look like be?

It, it

Are there any people?

No

Is there any people in
the boat?

No

Is the dog on?

No

What time of day is it?

It is
light

Can you tell the breed
of it?

No

What color is the boat?

White

What is the color?

It is

How it is like is?

Any there see the
driver?

No

Is it a Harley?
Any other?

What color is the man?

White

Is this a color photo?

Yes

What is the color?

Is

What color is his hair?

Black

What color is the
tennis?

It

Is the player?

Yes

How old is he?

Thirtys

Are there any other
people in the?

No

Is it a?

Yes

Is he have other?

Yes

Can you see any tennis?

No

Is it outdoors?

Yes

Is it sunny?

Yes

Can you see any people
in the?

No

Is there people?

No

Can you see any?

Yes

What time of day is it?

It

Is he playing?

Yes

Yes

Is the appear like be
be?

It

Is it a?

Yes

Is it sunny?

Yes

Are there any other in
the the?

No

Is it a?

Yes

Is it sunny?

Yes

Can you see the time?

No

Can you see the net?

No

Can you see sky?

No

Can you see the season?

No

Is it man?

Yes

Is it court?

A person is holding a ball
and tennis racket

What kind of dog
is it?

Is it big?

Figure 10: Diverse two-way dialogue generations from the BAR 10 model (block evaluations) for the 2 VD task – continued

