arXiv:1705.00464v2 [cs.CL] 16 Sep 2017

Speech-Based Visual Question Answering
Ted Zhang

Dengxin Dai

Tinne Tuytelaars

KU Leuven
tedz.cs@gmail.com

ETH Zurich
dai@vision.ee.ethz.ch

KU Leuven
tinne.tuytelaars@esat.kuleuven.be

Marie-Francine Moens

Luc Van Gool

KU Leuven
sien.moens@cs.kuleuven.be

ETH Zurich, KU Leuven
vangool@vision.ee.ethz.ch

ABSTRACT
This paper introduces speech-based visual question answering (VQA),
the task of generating an answer given an image and a spoken question. Two methods are studied: an end-to-end, deep neural network
that directly uses audio waveforms as input versus a pipelined approach that performs ASR (Automatic Speech Recognition) on the
question, followed by text-based visual question answering. Furthermore, we investigate the robustness of both methods by injecting
various levels of noise into the spoken question and find both methods to be tolerate noise at similar levels.

1

ASR

TextMod

pizza

what food is this?

INTRODUCTION

The recent years have witnessed great advances in computer vision,
natural language processing, and speech recognition thanks to the
advances in deep learning [16] and abundance of data [23]. This
is evidenced not only by the surge of academic papers, but also
by the world-wide industry interests. The convincing successes
in these individual fields naturally raise the potentials of further
integration towards solutions to more general AI problems. Much
work has been done to integrate vision and language, resulting in
a wide collection of successful applications such as image/video
captioning [27], movie-to-book alignment [31], and visual question
answering (VQA) [3]. However, the importance of integrating vision
and speech has remained relatively unexplored.
Pertaining to practical applications, voice-user interface (VUI)
has become more commonplace, and people are increasingly taking
advantage of its characteristics; it is natural, hands-free, eyes-free,
far more mobile and faster than typing on certain devices [22]. As
many of our daily tasks are relevant to visual scenes, there is a strong
need to have a VUI to talk to pictures or videos directly, be it for
communication, cooperation, or guidance. Speech-based VQA can
be used to assist blind people in performing ordinary tasks, and to
dictate robotics in real visual scenes in a hand-free manner such as
clinical robotic surgery.
This work investigates the potential of integrating vision and
speech in the context of VQA. A spoken version of the VQA1.0
dataset is generated to study two different methods of speech-based
question answering. One method is an end-to-end approach based on
a deep neural network architecture, and the other uses an ASR to first
transcribe the text from the spoken question, as shown in Figure 1.
The former approach can be particularly useful for languages that
are not serviced by popular ASR systems, i.e. minor languages that
have scarce text-speech aligned training data.

SpeechMod

pizza

Figure 1: An example of speech-based visual question answering and the two method in this study. A spoken question what
food is this? is asked about the picture, and the system is expected to generate the answer pizza.
The main contributions of this paper are three-fold: 1) We introduce an end-to-end model that directly produces answers from
auditory input without transformations into intermediate pre-learned
representations, and compare this with the pipelined approach. 2)
We inspect the performance impact of having different levels of
background noise mixed with the original utterances. 3) We release
the speech dataset, roughly 200 hours of synthetic audio data and 1
hour of real speech data, to the public. 1 The emphasis of this paper
is not on achieving state of the art numbers on VQA, but rather on
exploring ways to address a new and challenging task.

2 RELATED WORKS
2.1 Visual Question Answering
The initial introduction of VQA into the AI community [6], [18] was
motivated by a desire to build intelligent systems that can understand
the world more holistically. In order to complete the task of VQA,
it was both necessary to understand a textual question and a visual
1 http://data.vision.ee.ethz.ch/daid/VQA/SpeechVQA.zip

scene. However, it was not until the introduction of VQA1.0 [3] that
the application took mainstream in the computer vision and natural
language processing (NLP) communities.
Recently, popular topics of exploration have been on the development of attention models. Attention models were popularized
by their success with the NLP community in machine translation
[5], and quickly demonstrated their efficacy in computer vision [19].
Within the context of visual question answering, attention mechanisms ‘show’ a model where to look when answering a question.
Stacked Attention Network [30] learns an attention mechanism based
on the of the question’s encoding to determine the salient regions
in an image. More sophisticated attention-centric models such as
[17, 20, 28] were since then developed.
Other points of research are based on the pooling mechanism
that combines the language component with the vision components.
Some use an element-wise multiplication [29, 30] to pool these
modalities, while [17] and [9] have shown much success in using
more complex methods. Our work differs from theirs in that we
aim not to improve the performance of VQA, but rather add a new
modality of input and introduce appropriate new methods.

2.2

Integration of Speech and Vision

The works also relevant to ours are those integrating speech and vision. Pixeltone [15] and Image spirit [7] are examples that use voice
commands to guide image processing and semantic segmentation.
There is also academic work [12, 13, 26] and an app [1] that use
speech to provide image descriptions. Their tasks and algorithms
are both different from ours. We study the potential of integrating
speech and vision in the context of VQA and aim to learn a joint
understanding of speech and vision. Those approaches, however,
use speech recognition for data collection or result refinement.
Our work also shares similarity with visual-grounded speech
understanding or recognition. The closest one in this vein is [11], in
which a deep model is learned with speeches about image captions
for speech-based image retrieval. In a broader context of integration
of sound and vision, Soundnet [4] transfers visual information into
sound representations, but this differs from our work because their
end goal is to label a sound, not to answer a question.

2.3

End-To-End Speech Recognition

In the past decade, deep learning has allowed many fields in artificial
intelligence to replace traditional hand-crafted features and pipeline
systems with end-to-end models. Since speech recognition is typically thought of as a sequence to sequence transduction problem,
i.e. given an input sequence, predict an output sequence, the application of LSTM and the CTC [8, 10] promptly showed the success
needed to justify its superiority over traditional methods. Current
state of the art ASR systems such as DeepSpeech2 [2] uses stacked
Bi-directional Recurrent Neural Networks in conjunction with Convolutional Neural networks. Our model is similar to theirs in that
we use CNNs connected with an LSTM to process audio inputs,
however our goal is question answering and not speech recognition.

3

MODEL

Two models are employed in this work, they will be referred to
henceforth as TextMod and SpeechMod. TextMod and SpeechMod

Audio
Conv Layers
(Dimensions
in Table 1)

Text
Embedding
Image

LSTM

Image

LSTM

Dense

Dense

Dense

Dense

.

.

Dense

Dense

Dense

Dense

output

output

Figure 2: TextMod (left) and SpeechMod (right) architectures

only differ in their language components, keeping rest of the architecture the same. On the language side, TextMod takes as input a
series of one-hot encodings, followed by an embedding layer that
is learned from scratch, a LSTM encoder, and a dense layer. It is
similar to VQA1.0 with some minor adjustments.
The language side of SpeechMod takes as input the raw waveform, and pushes it through a series of 1D convolutions. After
the CNN layers follows a LSTM. The LSTM serves the same purpose as in TextMod, which is to interpret and encode the sequence
meaningfully into a single vector.
Convolution layers are used to encode waveforms because they
reduce dimensionality of data while finding salient patterns. The
maximum length of a spoken question in our dataset is 6.7 seconds
and corresponds to a waveform length of 107,360 elements, while
the minimum is 0.63 seconds and corresponds to 10,080 elements.
One could directly feed the input waveform to a LSTM, but a LSTM
will be unable to learn from sequences that are excessively long, so
dimensionality reduction is a necessity. Each consecutive convolution layer halves in filter length but doubles the number of filters.
This is done for simplicity rather than for performance optimization.
The main consideration taken in choosing the parameters is that
the last convolution should output dimensions of (x, 512), where x
must be a positive integer. x represents the length of a sequence of
512-dim vectors. The sequence is then fed into an LSTM, which
then outputs a single vector of (512). Thus, x should not be too big,
and the CNN parameters are chosen to ensure a sensible sequence
length. The exact dimensions of the convolution layers are shown
in Table 1. The longest and shortest waveforms correspond to final
convolution outputs of size (13, 512) and (1, 512) respectively. 512
is used as the dimension of the LSTM to be consistent with TextMod
and the original VQA baseline.
On the visual side, both models ingest as input the 4,096 dimensional vector of the last layer of VGG19 [25] followed by a single
dense layer. After both visual and linguistic representations are computed, they are merged using element-wise multiplication, a dense
layer, and an output layer. The full architecture of both these models

Table 1: Dimensions for the conv layers. Example shown with a 2 second long audio waveform, sampled at 16 kHz. The final output
dimensions are (3, 512)
Layer
Input Dim
# Filters
Filter Length
Stride
Output Dim

conv1
32,000
32
64
2
16,000

pool1
16,000
32
4
4
4,000

conv2
4,000
64
32
2
2,000

are seen in Figure 2, where is the symbol for element-wise multiplication. After merging the language and visual components of each
model, two dense layers are stacked. The last dense layer outputs a
probability distribution over the number of output classes, and the
answer corresponding to the element with the highest probability is
selected.
The architectures presented in this chapter were chosen for two
main reasons: simplicity and similarity. First, the intention is to
keep the model complexity low. In order to establish a baseline
for speech-based VQA, it is necessary to use only the bare minimum components. TextMod, as mentioned before, is similar to the
original VQA baseline, which is well referenced and remains the simplest architecture on VQA1.0. Despite its many convolution layers,
SpeechMod also uses minimal components. Second, it is important
that TextMod and SpeechMod differ from each other as little as
possible. Similarity between models allows one to locate the source
of discrepancies and helps produce a more rigorous comparison.
The only difference in the two models is replacing an embedding
layer with a series of convolution layers. In our implementation, the
layers that are common between the two models also have the same
dimensions.

4

conv3
500
128
16
2
250

pool3
250
128
4
4
62

conv4
62
256
8
2
31

pool4
31
256
4
4
7

conv5
7
512
4
2
3

0% Noise

what time of day is it

what is the number of the bus

are there clouds on the sky

what time of day isn’t

what is the number of the boss

are there clouds on sky

what time and day isn’t

where is it

are there files on the sky

what time of day is it

what is the number of the boss

are there clouds on the sky

30% Noise

50% Noise

DATA

We chose to use VQA1.0 Open-Ended dataset, for its numerous training examples and familiarity to those working in question answering.
To avoid confusion, VQA1.0 henceforth refers to the dataset and
the original paper, while VQA refers to the task of visual question
answering. The dataset contains 248,349 questions in the training
set, 121,512 in validation set, and 60,864 in the test-dev set. The
complete test set contains 244,302 questions, but because the evaluation server allows for only one submission, we instead evaluate on
test-dev, which has no such limit. During training, questions which
do not contain the 1000 most common answers are filtered out.
Amazon Polly API is used to generate audio files for each question. 2 The generated speech is in mp3 format, then sampled into
waveform format at 16 kHz. 16 kHz was chosen due to its common
usage among the speech community, but also because the model
used to transcribe speech was trained on 16 kHz audio waveforms.
It is worthwhile to note that the audio generator uses a female voice,
thus the training and testing data are all with the same voice, except
for the examples weve recorded, which is covered below. The full
Amazon Polly speech dataset will be made available to the public.
The noise we mixed with the original speech files is selected
randomly from the Urban8K dataset [24]. This dataset contains 10
2 The

pool2
2,000
64
4
4
500

voice of Joanna is used: https://aws.amazon.com/polly/

Human

Figure 3: Spectrograms for 3 example questions with corresponding transcribed text below. 3 synthetically generated and
1 human-recorded audio clips for each question.
categories: air conditioner, car horn, children playing, dog bark,
drilling, engine idling, gun shot, jackhammer, siren, and street music.
Some clips are soft enough in volume and thus considered background noise, others are loud enough to be considered foreground
noise. For each original audio file, a random noise file is selected,
and combined to produce a corrupted question file according to the
weighting scheme:
Wcor rupt ed = (1 − NL) ∗ Wor iдinal + NL ∗ Wnoise
where NL is the noise level. The noise audio files are subsampled
to 16 kHz in order to match that of the original audio file, and is
clipped to also match the spoken question length. When the spoken

Table 2: Word Error Rate from Kaldi speech recognition
Noise (%)
0
10
20
30
40
50

WER (%)
8.46
12.37
17.77
25.41
35.15
47.90

question is longer than the noise file, the noise file is repeated until
its duration exceeds that of the spoken question. Both files are
normalized before being combined so that contributions are strictly
proportional to the noise level chosen. We choose 5 noise levels to
mix together: 10%-50%, at 10% intervals. Anything beyond 50% is
unrealistic. A visualization of different noise levels can be seen in
Figure 3 and its corresponding audio clips can be found online.3
We also make an additional, supplementary study of the practicality of speech-based VQA with real data. 1000 questions from the
val set were randomly selected and recorded with human speakers.
Two speakers (one male and one female) participated the recording
task. In total, 1/3 of the data is from a male speaker, the rest is from
a female speaker. Both speakers are graduate students who are not
native anglophones. The data was recorded in an office environment,
and there are various background noises in the audio clips as they
naturally occurred.

5 EXPERIMENTS
5.1 Preprocessing
For SpeechMod, the first preprocessing step is to scale each waveform to a range of [-256, 256], similar to the procedure from SoundNet [4]. There was no need to center each example around 0, as they
are already centered. Next, each batch of waveforms were padded
with 0 at the end to be of the same length.
For TextMod, the standard preprocessing steps from VQA1.0 were
followed. The procedure tokenizes each sentence and replaces it
with a number that corresponds to the word’s index. These number
indices are used as input, since the question will be fed to the model
as a sequence of one hot encodings. Because questions have different
lengths, the 0 index is used as padding for sequences that are too
short. The 0 index essentially causes the model to skip that position.
0 is also used for unseen tokens, which is especially useful when
dealing with out of vocabulary words during evaluation.

5.2

ASR

We use Kaldi [21] for ASR, due to its open-source codebase and
popularity with the speech research community. The model used in
this work is a DNN-HMM4 that has been pre-trained on assistant.ai
logs (essentially short commands), making it suitable for transcribing
short utterances such as the questions in VQA1.0. Other ASRs such
as wit.ai from Facebook, Cloud Speech from Google, and Bing
Speech Microsoft were tested but not used in the final experiments
because Kaldi achieved the lowest word error rates.

Word error rate (WER) is used to measure the accuracy of speech
to text transcriptions. WER is defined as follows:
WER = (S + D + I )/N
Where S is the number of substitutions, D is the number of deletions,
and I is the number of insertions. N is the total number of words in
the sentence being translated. Each transcribed question is compared
with the original; the results are shown in Table 2. WER is not
expected to be a perfect measure of transcription accuracy, since
some words are more essential to the meaning of a sentence than
other words. For example, missing the word dog in the sentence
what is the dog eating is more detrimental than missing the word
the, but we nevertheless employ it to convey a general notion of how
many words are understood by the ASR. Naturally the more noise
there is, the higher the word error rate becomes. Due to transcription
errors, there are resulting questions that contain words not seen in
the original datasets. These words, as mentioned above, are indexed
as 0 and are masked when fed into TextMod.

5.3

Implementation

Keras was used to run all experiments, with the Adam [14] optimizer for both architectures. No parameter tuning was done; default
Adam parameters are as follows: learning rate=0.001, beta1=0.9,
beta2=0.999, epsilon=1e-08, learning rate decay=0.0. Training
TextMod for 10 epochs on train + val takes roughly an hour on
a Nvidia Titan X GPU, and our best model was taken at 30 epochs.
Training SpeechMod for 10 epochs takes roughly 7 hours. The
reported model is taken at 30 epochs. The code is available to the
public.5

6

RESULTS

The goal of the main experiments were to observe how each model
performs with and without different levels of noise added. Results
are reported on test-dev, which corresponds to training on train +
val (Table 3). The standard format of reporting results from VQA1.0
is followed: All is the overall accuracy, Y/N is for questions with
yes or no as answers, Number is for questions that are answered by
counting, and Other covers the rest.
TextMod is trained on the original questions (OQ), with the best
performing model being selected. ASR is used on the 0-50% variants
to convert the audio question to text. Then, the selected model from
OQ is used to evaluate based on the transcribed text. Concretely, the
best performing model obtained on test-dev is used to evaluate the
transcribed variants of test-dev. Likewise, SpeechMod is first trained
on audio data with 0% noise, with the strongest model being selected.
The selected model is used to evaluate on the 10-50% variants of
the same data subset. Typically, the best model on val is used to
evaluate on test or another ‘unseen’ portion of the dataset. However
in these experiments, the noisy variants of the same datasets are
in fact unseen because the data for which the model is trained on
contains no noise. We show this in the zero-shot section of the paper.
Blind denotes no visual information, meaning it removes the
visual components while rest of the model stays the same. TextMod
Blind is trained and evaluated on the original questions. SpeechMod

3 https://soundcloud.com/sbvqa/sets/speechvqa
4 https://github.com/api-ai/api-ai-english-asr-model

5 https://github.com/zted/sbvqa

Table 3: Accuracy on test-dev with different levels of noise
added. (Higher is better)

60
SpeechMod
TextMod

SpeechMod

Blind
OQ
0%
10%
20%
30%
40%
50%
Blind
0%
10%
20%
30%
40%
50%

All
53.74
48.76
56.66
54.03
52.56
50.22
47.03
42.83
37.12
42.05
46.99
45.81
43.33
40.07
35.85
32.14

Y/N
78.94
78.20
78,89
75.47
74.06
71.16
67.31
62.35
25.42
70.85
67.87
67.29
65.88
64.15
61.47
59.33

Number
35.24
35.68
37.24
36.82
36.50
35.72
34.45
31.97
27.05
31.62
30.84
30.13
29.24
27.82
24.68
20.84

Other
36.42
26.59
42.07
39.62
37.85
35.64
32.56
28.64
23.77
19.84
32.82
31.03
27.28
22.28
16.52
11.50

TextMod Blind OQ

Accuracy (%)

Baseline
TextMod

SpeechMod Blind 0% Noise

40

30

0

10

20
30
Noise (%)

40

50

Figure 4: SpeechMod and TextMod performance with varying
amounts of added noise on test-dev. Blind counterparts are not
tested on different noise levels.

6.1
Blind is trained and evaluated on the 0% noise audio. Baseline is
from VQA1.0 using the model ‘LSTM Q+I’.
A graphical version of the table is shown in Figure 4. The constant
values of SpeechMod Blind and TextMod Blind are included to show
the noise level at which they perform better than their full model
counterparts. Examples of the two models answering questions from
the dataset are shown in Figure 6.
One might imagine SpeechMod to perform better because of its
direct optimization and end-to-end training solely for the task, yet
this hypothesis does not hold true. At 0% noise, TextMod achieves
7% higher accuracy than SpeechMod. As noise is added, both
models initially falter at similar rates, although their trends seem to
head towards convergence. This is expected since, since at 100%
noise the question would not be audible at all; it would be random
guessing, thus both methods would perform exactly the same.
Next, we compare TextMod and SpeechMod against their respective Blind models. The bias of questions in VQA1.0 is well
documented. Namely, if the question is understood, there is a good
chance of answering correctly without looking at the image (i.e.
blind guessing). For example, Y/N questions have the answer yes
more commonly than no, so the system should guess yes if a question
is identified to be a Y/N type. As a reference, always answering yes
yields a Y/N accuracy of 70.81% on test-dev. The bias is clearly
evident in both test-dev and val for TextMod and SpeechMod; the
Y/N section of Blind always performs better than that of the 0% data.
Therefore, Blind tells us how many questions are understood by
these two modes of linguistic inputs. When comparing the linguistic
only models with their complementary TextMod and SpeechMod,
one can be certain that performances falling below the linguistic
signifies that the model no longer understands the questions. Furthermore, perceiving the image and a noisy question becomes less
informative than understanding a clean question without an image.

50

Zero-Shot

In this section zero-shot (ZS) results are analyzed to further understand the behavior of both models. ZS in the context of VQA refers
to questions that were never seen in training. To get ZS data, we discard questions in val subset that appeared in the train subset, which
decreased the number of valid questions from 104,654 to 65,365.
Put differently, ZS is simply a subset of val.
The models were trained on the complete train, and best performing on the complete val were selected. Next, the models were
tested on the original and ZS datasets with noise injected (Table 4,
Figure 5). These experiments were not be performed on test-dev
because the ground truth from test-dev and test are withheld and
cannot be evaluated partially on the server.
As one would expect, ZS accuracies are worse than accuracies of
the entire set, since models tend to perform more poorly on unseen
data. TextMod performs 4% better on the complete dataset than on
the ZS, and SpeechMod on the complete dataset performs better by
7%. The performance gap decreases as more noise is added. At 50%
noise, the performance on ZS and the original data have practically
converged for both models. To the models, questions seen during
the training but with high amount of noise added are as foreign as
unseen questions.

6.2

Human Recordings

Finally, a small, supplementary test is run on non-synthetic, humanrecorded questions to see if the models would perform differently
on real-world audio inputs. 1000 samples were randomly selected
from val, and the best performing models from the ZS section were
used for evaluation. Table 5 shows the performance on the synthetic
and human-recorded versions of this subset.
Although it is clear that both models have difficulties handling
recorded questions, SpeechMod performs especially poorly. TextMod
on the synthetic dataset achieves similar accuracy as it does on val
and test-dev with 40% noise. SpeechMod however, gets similar

Table 4: Accuracy on zero-shot with different levels of noise
added. (Higher is better)

60
SpeechMod val
TextMod val

SpeechMod

OQ
0%
10%
20%
30%
40%
50%
0%
10%
20%
30%
40%
50%

All
49.41
46.41
45.23
43.30
40.85
37.79
34.41
37.01
36.52
35.47
34.08
32.12
29.88

Y/N
77.23
73.37
71.93
69.26
65.84
62.56
59.58
65.58
65.12
64.04
62.77
60.59
57.70

Number
31.18
30.64
30.32
29.63
28.55
26.10
21.50
23.19
22.83
22.29
21.45
19.94
18.20

Other
27.12
24.38
23.24
21.75
19.89
16.91
13.42
12.99
12.45
11.29
9.67
7.81
6.09

Table 5: Performance on 1000 human-recorded questions.

SpeechMod (Recorded)
SpeechMod (Synthetic)
TextMod (Recorded)
TextMod (Original Text)

All
21.46
42.69
41.66
53.09

Y/N
57.26
66.58
66.33
77.73

Number
0.77
32.31
35.69
41.54

Other
0.91
27.58
25.37
38.26

performance as the synthetic data with 50% noise on only the Y/N
questions, while it seems to understand none of the other question
types.

6.3

Discussion

As a modality, speech contains more information than text. In the
process of reducing the high-dimensional audio inputs to the lowdimensional class output label (i.e. the answer), the best performing
system must be that which extracts patterns most effectively.
TextMod relies heavily on the intermediate ASR system, which
is more complicated than the entire architecture of SpeechMod, as
the number of parameters one needs to learn for speech recognition
is also much greater. The Kaldi model has also been trained on
many times more data than contained in VQA1.0. The ASR serves to
filter out noise in high dimensions and extract meaningful patterns
in the form of text. In a sense, one can think of the ASR as a
feature extractor, with text being the salient feature and an explicit
intermediate standardization of data before the question answering
module.
Conversely, the only audio data SpeechMod learns from are the
questions in the dataset. It does not include any mechanisms that
explicitly learn semantics in a language, nor does it have intermediate
data standardization. Thus, the model may not extract the concept
of words from audio sounds. Whether or not forcing the system to
learn words (i.e. transcribing words in the question and answering
simultaneously) will be beneficial is left to future research, but it is
evident that data standardization is helpful for unseen data.
In ZS experiments, the gap in performance between the unseen
and full dataset with TextMod is much smaller than in SpeechMod
(4% vs 7%). A text-based system can still glimpse the meaning

TextMod ZS

Accuracy (%)

TextMod

SpeechMod ZS

50

40

30

0

10

20
30
Noise (%)

40

50

Figure 5: SpeechMod and TextMod performance with varying
amounts of added noise for zero-shot subset in reference to the
complete datasets.

of a question even if a word has never been seen, but from the
perspective of SpeechMod, new words represent entirely different
signal trajectories. Furthermore, audio inputs are continuous streams,
making it difficult to differentiate when the new words begin or end.
A similar effect is amplified in answering human-recorded questions. The synthetic audio sounds monotonous, disinterested, with
little silence between words while the human-recorded audio has
inflections, emphasis, accents, and pauses. An inspection of the
spectrograms (Figure 3) confirms this, as the synthetic waveforms
have vastly different audio signatures. Because SpeechMod has no
training data similar to the human-recorded samples, it is unable to
extract salient patterns. In comparison, the ASR removes most of
the variance in the input by standardizing the audio into a compact,
salient textual representation. From the perspective of TextMod,
the human-recorded questions is only slightly different than those
provided in training.
It is evident in our experiments that text-based VQA performs
better than speech-based, but bearing in mind the simple architecture and limited amount of training data, we believe the results of
SpeechMod merits further study into end-to-end methods.

6.4

Future Work

As alluded to in previous sections, there are a few research directions
that may yield interesting results. One straightforward approach to
improving the end-to-end model is by data augmentation. It is widely
accepted that effectiveness of neural architectures is data driven, so
training with noisy data and different speakers will make the model
more robust to inputs during run time. Just as many possibilities
exist in improving the architecture. One can add feature extractors,
attention mechanisms, GAN training, or any amalgamation of the
techniques in the deep learning mainstream. An interesting study
would be to enforce the prediction of the question while simultaneously learning to answer the question. Doing so may improve

how many people are in this
photo?

what is leaning against the
house?

what has been upcycled to
make lights?

what is the teddy bear sitting
on?

TextMod : 2
SpeechMod : 2

TextMod : tree
SpeechMod : chair

TextMod : bulb
SpeechMod : bulb

TextMod : chair
SpeechMod : yes

Figure 6: Example of speech-based VQA answers. Correct answers in blue and incorrect in red.
performance, but more importantly allows us to interpret the concepts learned by the neural network.
Another direction is to restrict the amount of training data available to both approaches to observe their learning efficiency. For
example, minor languages may not have a reliable ASR. One can
simulate a minor language by training an ASR with only the data
available in the training set, and comparing this approach with the
end-to-end method trained on the same amount of data.

7

CONCLUSION

We have proposed speech-based visual question answering and introduced two approaches that tackle this problem, one of which can
be trained end-to-end on audio inputs. Despite its simple architecture, the end-to-end method works well when the test data has audio
signatures comparable to its training data. Both methods suffered
performance decreases at similar rates when noise is introduced. A
pipelined method using an ASR tolerates varied inputs much better
because it normalizes the input variance into text before running the
VQA module. We release the speech dataset and invite the multimedia research community to explore the intersection of speech and
vision.

REFERENCES
[1] 2015. Smile - Smart Photo Annotation. https://play.google.com/store/apps/details?
id=com.neuromorphic.retinet.smile. (2015).
[2] Dario Amodei, Rishita Anubhai, Eric Battenberg, Carl Case, Jared Casper, Bryan
Catanzaro, Jingdong Chen, Mike Chrzanowski, Adam Coates, Greg Diamos,
and others. 2015. Deep speech 2: End-to-end speech recognition in english and
mandarin. arXiv preprint arXiv:1512.02595 (2015).
[3] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra,
C. Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual Question Answering.
In International Conference on Computer Vision (ICCV).
[4] Yusuf Aytar, Carl Vondrick, and Antonio Torralba. 2016. SoundNet: Learning
Sound Representations from Unlabeled Video. CoRR abs/1610.09001 (2016).
http://arxiv.org/abs/1610.09001
[5] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint
arXiv:1409.0473 (2014).
[6] Jeffrey P Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller,
Robert C Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual
White, and others. 2010. VizWiz: nearly real-time answers to visual questions. In
Proceedings of the 23nd annual ACM symposium on User interface software and
technology. ACM, 333–342.
[7] Ming-Ming Cheng, Shuai Zheng, Wen-Yan Lin, Vibhav Vineet, Paul Sturgess,
Nigel Crook, Niloy J. Mitra, and Philip Torr. 2014. ImageSpirit: Verbal Guided
Image Parsing. ACM Trans. Graph. 34, 1 (2014), 3:1–3:11.

[8] Santiago Fernández, Alex Graves, and Jürgen Schmidhuber. 2007. An Application
of Recurrent Neural Networks to Discriminative Keyword Spotting. Springer
Berlin Heidelberg, Berlin, Heidelberg, 220–229. DOI:https://doi.org/10.1007/
978-3-540-74695-9 23
[9] Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and
Marcus Rohrbach. 2016. Multimodal compact bilinear pooling for visual question
answering and visual grounding. arXiv preprint arXiv:1606.01847 (2016).
[10] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber.
2006. Connectionist temporal classification: labelling unsegmented sequence
data with recurrent neural networks. In Proceedings of the 23rd international
conference on Machine learning. ACM, 369–376.
[11] David Harwath, Antonio Torralba, and James Glass. 2016. Unsupervised Learning
of Spoken Language with Visual Context. In Advances in Neural Information
Processing Systems. 1858–1866.
[12] Timothy J. Hazen, Brennan Sherry, and Mark Adler. 2007. Speech-based annotation and retrieval of digital photographs. In INTERSPEECH.
[13] D.V. Kalashnikov, S. Mehrotra, Jie Xu, and N. Venkatasubramanian. 2011. A
Semantics-Based Approach for Speech Annotation of Images. IEEE Trans. Knowl.
Data Eng. 23, 9 (2011), 1373–1387.
[14] Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
[15] Gierad P Laput, Mira Dontcheva, Gregg Wilensky, Walter Chang, Aseem Agarwala, Jason Linder, and Eytan Adar. 2013. PixelTone: a multimodal interface for
image editing. In CHI.
[16] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. Nature
521, 7553 (2015), 436–444.
[17] Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. 2016. Hierarchical
Question-Image Co-Attention for Visual Question Answering. (2016).
[18] Mateusz Malinowski and Mario Fritz. 2014. A multi-world approach to question
answering about real-world scenes based on uncertain input. In Advances in
Neural Information Processing Systems. 1682–1690.
[19] Volodymyr Mnih, Nicolas Heess, Alex Graves, and others. 2014. Recurrent
models of visual attention. In Advances in neural information processing systems.
2204–2212.
[20] Hyeonseob Nam, Jung-Woo Ha, and Jeonghee Kim. 2016. Dual Attention Networks for Multimodal Reasoning and Matching. CoRR abs/1611.00471 (2016).
[21] Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek,
Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz,
Jan Silovsky, Georg Stemmer, and Karel Vesely. 2011. The Kaldi Speech Recognition Toolkit. In IEEE 2011 Workshop on Automatic Speech Recognition and
Understanding. IEEE Signal Processing Society. IEEE Catalog No.: CFP11SRWUSB.
[22] Sherry Ruan, Jacob O Wobbrock, Kenny Liou, Andrew Ng, and James Landay.
2016. Speech Is 3x Faster than Typing for English and Mandarin Text Entry on
Mobile Devices. arXiv preprint arXiv:1608.07323 (2016).
[23] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean
Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, and
others. 2015. Imagenet large scale visual recognition challenge. International
Journal of Computer Vision 115, 3 (2015), 211–252.
[24] Justin Salamon, Christopher Jacoby, and Juan Pablo Bello. 2014. A Dataset and
Taxonomy for Urban Sound Research. In Proceedings of the ACM International
Conference on Multimedia, MM ’14, Orlando, FL, USA, November 03 - 07, 2014.
1041–1044. DOI:https://doi.org/10.1145/2647868.2655045

[25] K. Simonyan and A. Zisserman. 2015. Very Deep Convolutional Networks for
Large-Scale Image Recognition. In ICLR.
[26] R.K. Srihari and Zhongfei Zhang. 2000. Show&Tell: a semi-automated image
annotation system. MultiMedia, IEEE 7, 3 (2000), 61–71.
[27] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show
and Tell: A Neural Image Caption Generator. In CVPR.
[28] Caiming Xiong, Stephen Merity, and Richard Socher. 2016. Dynamic memory
networks for visual and textual question answering. arXiv 1603 (2016).
[29] Huijuan Xu and Kate Saenko. 2016. Ask, attend and answer: Exploring questionguided spatial attention for visual question answering. In European Conference
on Computer Vision. Springer, 451–466.
[30] Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. 2016.
Stacked attention networks for image question answering. In Proceedings of the
IEEE Conference on Computer Vision and Pattern Recognition. 21–29.
[31] Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun,
Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards
story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE International Conference on Computer Vision. 19–27.

