arXiv:1710.09798v1 [cs.CV] 26 Oct 2017

Lip2AudSpec: Speech reconstruction from silent lip
movements video
Hassan Akbari
Department of Electrical Engineering
Columbia University, New York, NY, USA
ha2436@columbia.edu

Himani Arora
Department of Electrical Engineering
Columbia University, New York, NY, USA
ha2434@columbia.edu

Liangliang Cao
Department of Electrical Engineering
Columbia University, New York, NY, USA
lc2922@columbia.edu

Nima Mesgarani
Department of Electrical Engineering
Columbia University, New York, NY, USA
nima@ee.columbia.edu

Abstract
In this study, we propose a deep neural network for reconstructing intelligible
speech from silent lip movement videos. We use auditory spectrogram as spectral
representation of speech and its corresponding sound generation method resulting
in a more natural sounding reconstructed speech. Our proposed network consists of
an autoencoder to extract bottleneck features from the auditory spectrogram which
is then used as target to our main lip reading network comprising of CNN, LSTM
and fully connected layers. Our experiments show that the autoencoder is able
to reconstruct the original auditory spectrogram with a 98% correlation and also
improves the quality of reconstructed speech from the main lip reading network.
Our model, trained jointly on different speakers is able to extract individual speaker
characteristics and gives promising results of reconstructing intelligible speech
with superior word recognition accuracy.

1

Introduction

A phoneme is the smallest detectable unit of a (spoken) language and is produced by a combination
of movements of the lips, teeth and tongue of the speaker. However, some of these phonemes are
produced from within the mouth and throat and thus, cannot be detected by just looking at a speaker’s
lips. It is for this reason that the number of visually distinctive units or visemes is much smaller than
the number of phoneme making lip reading an inherently difficult task. For example, it is nearly
impossible to visually distinguish ’s’ from ’z’ as they are both pronounced with tongue near the gums
and differ by the presence of voicing or vibration of the vocal cords in one (in the case of ’z’) as
opposed to the other (here, ’s’).
In the past, most works have viewed lip reading as a video to text task where the final output is a set
of textual sentences corresponding to the lip movements of the speaker. We go one step further by
modeling it as a video to speech task where our output is the acoustic speech signal allowing us to
recover not just the information but also the style of articulation.
This opens an entirely new world of applications in the audio-visual domain - improving audio in
existing videos in the sequences where someone is talking, for example blogging videos or news
anchoring videos, enabling video-chatting in silent areas like libraries or in noisy environments.
The challenge of speech generation suggested in related study is that speech models often lose pitch
information and consequently have low intelligibility in the reconstructed speech. In this paper, we
propose an end-to-end learning network to learn the high dimensional auditory spectrogram with its
corresponding audio-generation process Chi et al. (2005). Compared with classical models including

LPC, LSP, traditional spectrogram, auditory spectrogram preserves both the pitch and resonance
information of audio signals, by a 128 dimensional feature at every time step. If the audio is of 8kHz,
auditory spectrogram will yield to millions of parameters. Moreover, these auditory spectrogram are
highly correlated, which makes it is very difficult for the network to learn the auditory spectrogram
directly.
In this paper, we developed two techniques which work well for these tasks. First, we train a deep
autoencoder network with additive Gaussian noise, which can robustly recover original spectrogram
with a low dimensional sparse feature vector even in noise. Second, we employ a new cost as a
combination of mean squared loss and correlation loss which preserves both the shape and relative
amplitude of these audio features.
Based on the deep autoencoder of audio spectrogram, we build a CNN + LSTM network to construct
the speech for lip videos. This allows us to generate an acoustic signal that is much more natural
sounding than the previous approaches. Our experiments show that the original speech signal can be
reconstructed from the bottleneck features of an auditory spectrogram with a 99.3% correlation and
without any significant loss to quality. We believe that this sound generation model with auditory
spectrogram representation is superior than traditional spectrogram magnitude based models, and has
a lot of potentials for future applications.

2

Related Works

Lip reading has traditionally been posed as a classification task where words or short phrases from
a limited dictionary are classified based on features extracted from lip movements. Some of the
early works such as by Ngiam et al. (2011), Petridis and Pantic (2016) and Noda et al. (2015) used
a combination of deep learning and hand-crafted features in the first stage followed by a classifier.
More recently there has been a surge in end-to-end deep learning approaches for lip reading. Wand
et al. (2016), Assael et al. (2016) and Chung and Zisserman (2016) focused on either word level or
sentence-level prediction using a combination of convolutional and recurrent networks.
Our proposed network also follows a similar structure in the sense that the initial layers consist of
convolutional layers to extract features from the video followed by an LSTM to model the temporal
dependencies. However, we model our output as a generative task over the audio frequency space to
directly produce the corresponding speech signal at every time-step. We describe here two works that
are most closest to ours.
Milner and Le Cornu (2015) reconstructed audio from video, by estimating the spectral envelope using
a deep-learning network composed solely of fully connected layers and trained on hand-engineered
visual features obtained from mouth region. This approach had the limitation of missing certain
speech components such as fundamental frequency and aperiodicity which was then determined
artificially thereby compromising quality in order to maximize intelligibility. Ephrat and Peleg (2017)
modified this technique by using an end-to-end CNN to extract visual features from the entire face
while applying a similar approach for modeling audio features using 8th order Linear Predictive
Coding (LPC) analysis followed by Line Spectrum Pairs (LSP) decomposition. However, it also
suffered from the same missing excitation parameters resulting in an unnatural sounding voice. In a
different setting from lip reading, Owens et al. (2016) also synthesized sound from silent videos of
people hitting and prodding objects with a drumstick. They used a CNN to compute features from
spacetime images which can be thought of as 3D video CNNs which were fed to a recurrent neural
network (RNN). The output of this network was a 10 dimensional representation of the original
42-dimensional spectral envelope projected using PCA.
Our sound generation model differs from these as we use a specific spectrogram inspired by human
auditory cortex whose re-synthesis to speech is of a higher quality than traditional spectrograms.
This allows us to generate an acoustic signal that is much more natural sounding than the previous
approaches. However, the spectrograms in itself are highly correlated and usually difficult for the
networks to learn accurately. To bypass this issue, we designed a deep autoencoder to extract
compressed features of the spectrogram and forms the target for the main lip reading network. In the
sequel, we will explain our proposed methods for this aim and will elaborately evaluate them and
compare the results with the best baseline.
2

3
3.1

Proposed Networks
Data Preparation

Data preparation in this study is really important. We first converted each lip movement video to
grayscale and normalized it to have zero mean and unit standard deviation. The face region was
then extracted from each frame and resized to have dimensions WxH. It was then divided into K
non-overlapping slices each of length Lv . First and second order temporal derivatives at each frame
were calculated to form a 4D tensor of shape (3, H, W, Lv ), where 3 is the number of time-derivative
channels (0th , 1st and 2nd order). The target bottleneck feature vector was also divided into K slices
with length La and no overlap.
3.2

Network I: Autoencoder

The original audio waveform was first downsampled to 8kHz and converted into its auditory spectrogram representation with 128 frequency bins. In order to compress the auditory features, we designed
a deep autoencoder network. It consists of an input layer of size 128 followed by a dense layer of
512 neurons to extract enough features from the input. The subsequent hidden layers then follow the
typical autoencoder structure with the number of hidden units decreasing initially from 128 to 64 until
the bottleneck layer of size 32 (which was found to be most optimal as shown in the experiments).
After this, the structure is mirrored on the other side of the bottleneck (decoder part) to have a hidden
layer of 64 neurons and then finally an output layer of the same size as the input layer. Thus, both the
input and output of the network is the 128 frequency bin auditory spectrogram. In addition to this,
the output of the activation of the bottleneck is contaminated with Gaussian noise. Our experiments
show that this improves the robustness of the decoder network. All the non-linearities throughout the
autoencoder are LeakyRelu [Maas et al. (2013)] except the bottleneck for which we use sigmoid. A
summary of the proposed autoencoder can be seen in Table 1.
3.3

Network II: Lip Reading Network

The input to the main lip reading network is the pre-processed video slice reshaped as a tensor of shape
(None, 3, H, W, Lv ) where 3 is the number of time-derivative channels, H and W is the height and
width of the cropped face images respectively and Lv is the number of video slices. Spatiotemporal
features of the video sequence are extracted using a 7-layer 3D convolutional network described in
Table 1. The output of each convolution has the same size as input and is followed by a 3D max
pooling layer. All throughout the CNN, the temporal dimensions and order is maintained to enable
using an LSTM to model the time-dependencies. Because of this, the output of the convolutional
network block is reshaped to a tensor of shape (Lv , Nf ) in which Nf represent the spatial features
extracted by the convolutional network. This reshaped tensor is fed into a single-layer LSTM network
with 512 units to capture the temporal pattern. Output of this layer is further flattened and fed into a
single-layer fully connected network and then finally to the output layer. The output layer has 32×La
units to give the 32-bin×La -length bottleneck features which is then connected to the decoder part
of the pre-trained autoencoder to reconstruct the auditory spectrogram. The overall structure of the
proposed network cana be seen in Figure 1.
Non-linearity for all hidden layers of the convolutional network block is LeakyReLU, whereas its
output, LSTM and MLP all have ELU [Clevert et al. (2015)] as the non-linearity. However, the
final output layer has sigmoid non-linearity to match the non-linearity at the bottleneck layer of
the autoencoder. The audio waveform is reconstructed using the auditory spectrogram toolbox
developed by Chi et al. (2005) (which estimates the phase corresponding to magnitude of the auditory
spectrogram and performs the inverse transform to reconstruct the audio waveform).

4

Experiments

In this section, we elaborate the details about implementation of the network and provide detailed
information regarding the evaluation.
3

LSTM

Flattened Data
Decoder part of the
pre-trained auto-encoder

Dense

Dense

4096

32×20

Reshaped Output

Reashape

512

32

20

Samples

2560

Dense (128)

5

Input (32)

Reashape
(5,128×4×2)

Flatten

Dense (64)

7-layer Conv3D

Temporal features

Input

Spatial Features
Frequency bins

Figure 1: The overall structure for the proposed network. The network gets a video sequence and
captures spatiotemporal features and generates coded features of the audio corresponding to that
video. Those reconstructed features are then decoded using the pre-trained autoencoder.

Table 1: Structure of the proposed networks
Autoencoder
Layers

Size

Layers

Size

Input layer

(None, 128)

Input layer

(None, 3, 128, 128, 5)

Dense (512)
LeakyReLU

(None, 512)

Conv3D (32)
LeakyReLU
MaxPooling3D (2,2,1)

(None, 32, 64, 64, 5)

(None, 128)

Conv3D (32)
LeakyReLU
MaxPooling3D (2,2,1)

(None, 32, 32 32, 5)

(None, 64)

Conv3D (32)
LeakyReLU
MaxPooling3D (2,2,1)

(None, 32, 16 16, 5)

Conv3D (64)
LeakyReLU

(None, 64, 16, 16, 5)

Conv3D (64)
LeakyReLU
MaxPooling3D (2,2,1)

(None, 64, 8, 8, 5)

Conv3D (128)
LeakyReLU

(None, 128, 8, 8, 5)

Conv3D (128)
ELU (alpha=1.0)
MaxPooling3D (2,2,1)

(None, 128, 4, 4, 5)

Dense (128)
LeakyReLU
Dense (64)
LeakyReLU
Dense (32)
Sigmoid

(None, 32)

Additive Gaussian Noise
(Sigma=0.05)
Dense (64)
LeakyReLU
Dense (128)
LeakyReLU

4.1

Convolutional network block

(None, 32)

(None, 64)
(None, 128)

Dataset

The dataset used for training the network was the GRID audio-visual corpus [Cooke et al. (2006)]
which consists of audio and video recordings of 34 different speakers (male and female). For each
speaker, there are 1000 utterances and each utterance is a combination of six words from a 51-word
vocabulary (shown in Table 2). Videos and audios are both 3 seconds long with a sampling rate of
25fps and 44kHz respectively. They are pre-processed as described in section 3.1 and then fed to the
networks. We conducted our training and evaluation using videos from two male speakers (S1, S2)
and two female speakers (S4, S29).

Command
bin
lay
place
set

Table 2: GRID vocabulary
Color Preposition
Letter
blue
green
red
white

at
by
in
with

A-Z
minus W

4

Digit

Adverb

0-9

again
now
please
soon

4.2

Implementation

As mentioned before, the pre-processed video was used as input to the network. We fixed the length
of each video slice Lv to 5 (which is equivalent to 200 ms), and length of each audio slice La = 20
was also set accordingly such that the number of audio and video slices K are equal. We found that
both width W and height H of cropped face images when set to 128 was sufficient to extract enough
features. We shuffled all the utterances from all 4 speakers and selected 10 samples for validation, 10
for testing and the rest for training.
We used Keras [Chollet et al. (2015)] with Tensorflow backend [Abadi et al. (2015)] for implementing
the network. Initialization of the network weights was performed using the proposed method by
He et al. (2015). We used batch normalization [Ioffe and Szegedy (2015)] for all layers, dropout
[Srivastava et al. (2014)] of p=0.25 every two layers in convolutional block and L2 penalty multiplier
set to 0.0005 for all convolutional layers. For LSTM and MLPs after convolutional blocks, we
used dropout of p=0.3 and regularization was not used. We first trained the autoencoder on the 128
frequency bin auditory spectrogram of the training audio samples with a mini-batch size of 128. After
training, we extracted the 32-bin bottleneck features which we then provided as target features for the
main network. The main lip reading network was trained using a batch size of 32 and the parameter
α for ELU non-linearity was set as 1. To improve robustness of the network, we performed data
augmentation in each epoch by randomly selecting videos and either flipping them horizontally or
adding small Gaussian noise. Optimization was performed using Adam [Kingma and Ba (2014)]
with an initial learning rate of 0.0001, which was reduced by a factor of 5 if validation loss was not
improving in 4 consecutive epochs. The loss function we used for all our networks was a combination
of mean squared error (MSE) and correlation as given by:
P
¯
(yi − ȳ)(ŷi − ŷ)
1X
(yi − ŷi )2 − q P i
λ
(1)
P
n i
¯ 2)
( i (yi − ȳ)2 )( i (ŷi − ŷ)
in which, λ is the hyper-parameter for controlling balance between the two loss functions. We show
that this loss function (that we call it CorrMSE ), with λ fixed to 1, performs better than both MSE
and correlation. For auditory spectrogram generation, we used NSRtools, a Matlab toolbox for
proposed method by Chi et al. (2005). Parameters for all auditory spectrogram generation and audio
waveform reconstruction from the spectrogram were frm_len = 10, tc = 10, fac = −2, shft = −1.
It’s worth mentioning that for extracting better features using the autoencoder, we first compressed
the spectrograms by raising them to the power of 1/3 and for retrieving the audio waveform from
the output of the decoder we reversed the compression process by raising it to the power of 3 before
feeding it to the auditory spectrogram toolbox for waveform reconstruction.
4.3

Results

In this section, we first discuss the ability of the autoencoder for coding an auditory spectrogram
with 128 frequency bins to N-bin bottleneck features (N<128), then we discuss the necessity of using
the autoencoder in the proposed structure and then we study the quality of reconstructed signals. In
the end, performance of the proposed method will be compared to the baselines using both human
evaluation and quantitative quality measures.
4.3.1

Quality and accuracy measurement of the reconstructions

For evaluating both the autoencoder and the main network, we first reconstruct the auditory spectrogram and measure 2D correlation between the reconstructed (Ŷ) and the main spectrogram (Y) using
2D Pearson’s correlation coefficient. Then, the audio waveforms corresponding to the spectrograms
are recovered using the method in Chi et al. (2005), and quality of the reconstructed audio is measured
using the standard Perceptual Evaluation of Speech Quality (PESQ), an automated assessment of the
speech quality Rix et al. (2001). Thus, in all evaluations we use an accuracy measure in frequency
domain (Corr2D) and a quality measure in time domain (PESQ). We also measure intelligibility of
the final reconstructions using Spectro Temporal Modulation Index (STMI) Elhilali et al. (2003).
4.3.2

Autoencoder

We trained the autoencoder model on 90% of the spectrograms from the GRID corpus for four
speakers, S1 (male), S2 (male), S4 (female) and S29 (female), used 5% for validation during the
5

training and tested it on the remaining 5% which where completely unseen. We used CorrMSE as
loss function and the training was done on 50 epochs and validation loss stopped improving after 40
epochs. In our evaluations on the proposed autoencoder, we examine effect of 1. numbers for the
bottleneck nodes, 2. additive noise to the bottleneck, 3. dropout on quality of decoded audio. It’s
worth mentioning that except the study on number of bottleneck nodes, the rest of conditions are with
bottleneck_nodes=32. Table 3 shows the Corr2D and PESQ measures for the mentioned conditions.
Table 3: Quality and accuracy measures for coded-decoded audio using the proposed autoencoder
Measure 16 nodes 32 nodes 64 nodes No noise (32 nodes) Dropout (32 nodes)
PESQ
Corr2D

2.76
0.98

2.81
0.98

2.92
0.99

2.88
0.97

2.33
0.95

It is clear that by increasing number of nodes, both correlation and quality improve. Also, using
dropout makes the results worse, and by not using noise we lose the accuracy of reconstruction,
however it improves the quality. A visual evaluation of the autoencoder (32 nodes, additive noise) can
also be seen in Figure 2. As it can be seen, the proposed autoencoder is able to code a spectrogram
to 4 times smaller feature space and decode it to the original spectrogram space with a high accuracy
even in noisy conditions. This robustness makes it a good choice for being used as target to the main
lip-reading network instead of the original spectrogram, since it can handle the unwanted noise or
variations in the input videos to the lip-reading network and consequently, variations in the output of
it. In section 4.3.5 we more discuss its effect on the overall result in main lip-reading network.
Bottleneck features

Original
120

Coded-decoded spectrogram, Corr2D=0.98334

30

120

20

80

10

40

100
60
20

100
80

0.56

60

1.12

1.68

2.24

0.56

Bottleneck features + Gaussian noise

40

1.68

2.24

120

30

20

1.12

Coded-decoded spectrogram, Corr2D=0.96716
100

0.56

1.12

1.68

2.24

20

80

10

40

60
20
0.56

1.12

1.68

0.56

2.24

1.12

1.68

2.24

Figure 2: Reconstructed audio spectrograms

4.3.3

Lip-reading network

We trained the main lip-reading network with 150 epochs on pre-processed train set (train-validationtest had the same combination as autoencoder’s). Validation accuracy stopped improving after
around 100 epochs. For a comparison, we also trained-validated-tested the best baseline (up to our
knowledge), Vid2Speech Ephrat and Peleg (2017), on the same train-validation-test. We conducted
two different class of evaluations on the results of the two methods: 1. Computer-based evaluation,
2. Human evaluation. For the first one we used STMI, PESQ and Corr2D measures discussed in
Section 4.3.1. Average STMI, PESQ and Corr2D for the two methods can be found in Table 4.
As it can be seen from the table, the reconstructed audios using the proposed method have both
Table 4: Quality and accuracy measures for our proposed method compared to Vid2Speech
Measure
Method
S1
S2
S4
S29 Average
STMI

Our method
Vid2Speech

0.82
0.58

0.84
0.59

0.84
0.46

0.82
0.48

0.80
0.52

PESQ

Our method
Vid2Speech

2.07
1.90

2.01
1.74

1.61
1.79

1.84
1.62

1.88
1.76

Corr2D

Our method
Vid2Speech

0.89
0.62

0.88
0.52

0.88
0.64

0.87
0.65

0.88
0.61

6

higher quality and intelligibility compared to the baseline. Our method has also been able to get
a significantly higher Corr2D compared to Vid2Speech which indicates the high reconstruction
accuracy, specifically the pitch information, in frequency space. To shed light on that, we generated
spectrograms for a random sample from the test set and compared both reconstructions with the
original spectrogram of that sample, which can be seen in Figure 3.
Original

120

Our method, Corr2D=0.8644

120
100

100

80

80

80

60

60

60

40

40

40

20

20
0.56

1.12

1.68

2.24

2.8

0.56

1.12

1.68

2.24

20
0.56

2.8

Vid2Speech, Corr2D=0.32293

120

100

1.12

1.68

2.24

2.8

0.56

1.12

1.68

2.24

2.8

0.56

1.12

1.68

2.24

2.8

0.56

1.12

1.68

2.24

2.8

Figure 3: Reconstructed audio spectrograms
It is clear from the figure that accuracy of audio reconstruction in frequency space is much more
accurate than the baseline. It’s also worth mentioning that the model can successfully handle
connections between windows and memorizes the time pattern for continuing a phoneme from one
window sample to another. Although, both methods fail in retrieving high frequency information
which is mainly due to the nature of the task. Not all phonemes can be recovered from lip-movements
alone, since valuable information about speech that is mostly shaped by tongue movements is missing
in most of the articulations. For example, although many of the consonants can be recovered solely
from lip movements, it is quite hard to reconstruct different vowels accurately.
4.3.4

Human evaluations

We conducted a survey on Amazon Mechanical Turk to evaluate the intelligibility and quality of
reconstructed speech by our method as well as ’Vid2Speech’. The task was to transcribe each
audio sample (human based speech recognition) for analyzing its intelligibility as well as to answer
questions for judging the quality of the audio. These questions asked the workers to rate each sample
on a scale of 1-5 on the basis of how natural it sounded with 1 being unnatural and 5 being extremely
natural. In addition to this, workers also had to guess the gender of the speaker, in which they were
given 3 choices: male, female and hard to say. Each audio sample was evaluated by 20 unique workers
who were provided with the GRID vocabulary and were allowed to replay the audio unlimited times.
Table 5 shows the result of this evaluation.
Table 5: Speaker-wise intelligibility & qualitative assessments for audio-only
Ephrat and Peleg (2017)

Ours

Measure

S1

S2

S4

S29

Avg.

S1

S2

S4

S29

Avg.

Accuracy(%)
Natural sound (1-5)
Correct Gender (%)
Hard to say (%)

35.2
1.13
58.0
36.0

51.2
1.45
77.0
16.0

57.7
1.44
21.0
42.0

59.6
1.37
17.0
32.0

50.9
1.35
43.2
31.5

49.3
1.69
85.83
9.16

56.1
1.48
79.2
12.5

54.9
1.67
83.3
10.0

63.7
1.67
92.0
2.0

55.8
1.63
85.1
8.4

We can see that in terms of word accuracy, our model performs significantly better than Vid2Speech
for three out of four speakers. This leads us to the conclusion that the reconstructed speech by our
method has more intelligibility and improves the word recognition accuracy on the baseline by 5%
when averaged over all four speakers. The accuracy for random guessing is 19% here. In terms of
quality of speech, our method consistently outperforms Vid2Speech. Not only is our reconstructed
speech more natural, it also shows significant speaker dependent characteristics such as gender which
is due to correct pitch information retrieval. On the other hand, Vid2Speech suffers from lack of pitch
information which results in poor speech quality especially for female speakers.
We also summarized the category-wise word accuracies for the two methods and can be found in
Table 6.As expected, the accuracy for ’Letter’ is much lower than the other categories. This can
be attributed to the fact that alphabets are harder to infer mainly due to the inherent ambiguity in
lip-reading of phonemes that get mapped to the same visemes. However, the accuracy for words are
significantly higher and they are also better than the performance of Vid2Speech.
7

Table 6: Word category-wise accuracies for audio-only
Method
Command Color Preposition Letter Digit
Ephrat and Peleg (2017)
Ours
Random selection
4.3.5

58.7%
72.7%
25.0%

70.6%
85.7%
25.0%

47.8 %
48.8%
25.0%

9.1%
6.0%
4.0%

Adverb

46.2%
40.0%
10.0%

73.2%
81.8%
25.0

Necessity of using autoencoder and proper loss function

As a study on effect of different loss functions on the overall results, we compared results of the main
lipreading network trained using MSE, Corr2 and a combination of the two (CorrMSE). We also
conducted a test to see how the autoencoder helps the main network to have more accurate reconstructions. Thus, instead of using bottleneck features as the target we directly fed the spectrogram to
the output layer of the main lip-reading network. We chose one of the speakers (S29) which had a
wide frequency range and subtle high frequency information in its spectrograms, and evaluated the
methods in the mentioned conditions on it. Table 7 shows PESQ and Corr2D measures for both using
bottleneck features and spectrograms as target to the network using different loss functions.
Table 7: Quality measures for different loss functions and different target features
Measure

MSE(Bott.)

Corr(Bott.)

CorrMSE(Bott.)

MSE(Spec.)

Corr(Spec.)

CorrMSE(Spec.)

PESQ
Corr2D

1.54
0.84

1.73
0.88

1.76
0.89

1.29
0.87

1.69
0.89

1.58
0.88

It is clear that using MSE as loss function results in a relatively poor quality of overall reconstructed
speech whereas correlation and CorrMSE result in having both higher Corr2D (in frequency) and
PESQ (in time) for the speech reconstructions. CorrMSE performs worse than correlation in using
spectrogram, however using it as loss function while feeding bottleneck features as targets to the
network results in having the best performance among all the conditions. We also examined the effect
of three conditions (for the bottleneck nodes) on the overall lip-reading performance: 1. number
of bottleneck nodes, 2. feeding noise to the bottleneck nodes while training (the autoencoder), 3.
Using dropout while training (the autoencoder). Table 8 shows PESQ and Corr2D measures (on
reconstructed speech from lip movements) based on the mentioned conditions.
Table 8: Quality and accuracy measures while feeding bottleneck features as target
Autoencoder 16 nodes 32 nodes 64 nodes 32 nodes (w/o noise) 32 nodes (w/ dropout)
PESQ
Corr2D

1.19
0.89

1.76
0.89

1.26
0.87

1.09
0.46

1.29
0.88

Generally, it’s easier for the lip-reading network to reconstruct smaller number of features (as we
saw that their loss values were better). But, it’s harder for the autoencoder with smaller bottleneck to
recover the spectrogram from those features, and it’s the opposite for larger bottlenecks. Because of
this trade-off, it’s clear that there should be a point in the middle for having the balance. It’s also
clear from the table that using additive Gaussian noise significantly improves overall performance of
the lip-reading structure. Even using dropout cannot handle the variations in the output of the main
network and it fails in quality. Based on the results in this section, we used 32 nodes bottleneck with
additive noise for all our evaluations on the main network.

5

Conclusion

In this paper, we proposed a structure consisting of a deep autoencoder for coding speech and a
deep lip-reading network for extracting speech-related features from the face. We showed that such
a combination improves both quality and accuracy of the reconstructed audio. We also conducted
different tests for comparing our network with a strong baseline and showed that the proposed
structure outperforms the baseline in speech reconstruction. Future work is to collect more train
data, including emotions, and to propose an end-to-end structure to directly estimate raw waveform
from facial speech-related features. Codes and samples for this project are available online on:
https://github.com/hassanhub/LipReading
8

References
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin,
M., Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard, M., Jia, Y., Jozefowicz, R., Kaiser, L., Kudlur,
M., Levenberg, J., Mané, D., Monga, R., Moore, S., Murray, D., Olah, C., Schuster, M., Shlens, J., Steiner,
B., Sutskever, I., Talwar, K., Tucker, P., Vanhoucke, V., Vasudevan, V., Viégas, F., Vinyals, O., Warden, P.,
Wattenberg, M., Wicke, M., Yu, Y., and Zheng, X. (2015). TensorFlow: Large-scale machine learning on
heterogeneous systems. Software available from tensorflow.org.
Assael, Y. M., Shillingford, B., Whiteson, S., and de Freitas, N. (2016). Lipnet: Sentence-level lipreading. arXiv
preprint arXiv:1611.01599.
Chi, T., Ru, P., and Shamma, S. A. (2005). Multiresolution spectrotemporal analysis of complex sounds. The
Journal of the Acoustical Society of America, 118(2):887–906.
Chollet, F. et al. (2015). Keras. https://github.com/fchollet/keras.
Chung, J. S. and Zisserman, A. (2016). Lip reading in the wild. In Asian Conference on Computer Vision, pages
87–103. Springer.
Clevert, D.-A., Unterthiner, T., and Hochreiter, S. (2015). Fast and accurate deep network learning by exponential
linear units (elus). arXiv preprint arXiv:1511.07289.
Cooke, M., Barker, J., Cunningham, S., and Shao, X. (2006). An audio-visual corpus for speech perception and
automatic speech recognition. The Journal of the Acoustical Society of America, 120(5):2421–2424.
Elhilali, M., Chi, T., and Shamma, S. A. (2003). A spectro-temporal modulation index (stmi) for assessment of
speech intelligibility. Speech communication, 41(2):331–348.
Ephrat, A. and Peleg, S. (2017). Vid2speech: Speech reconstruction from silent video. arXiv preprint
arXiv:1701.00495.
He, K., Zhang, X., Ren, S., and Sun, J. (2015). Delving deep into rectifiers: Surpassing human-level performance
on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages
1026–1034.
Ioffe, S. and Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal
covariate shift. arXiv preprint arXiv:1502.03167.
Kingma, D. and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Maas, A. L., Hannun, A. Y., and Ng, A. Y. (2013). Rectifier nonlinearities improve neural network acoustic
models. In Proc. ICML, volume 30.
Milner, B. and Le Cornu, T. (2015). Reconstructing intelligible audio speech from visual speech features.
Interspeech 2015.
Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H., and Ng, A. Y. (2011). Multimodal deep learning. In
Proceedings of the 28th international conference on machine learning (ICML-11), pages 689–696.
Noda, K., Yamaguchi, Y., Nakadai, K., Okuno, H. G., and Ogata, T. (2015). Audio-visual speech recognition
using deep learning. Applied Intelligence, 42(4):722–737.
Owens, A., Isola, P., McDermott, J., Torralba, A., Adelson, E. H., and Freeman, W. T. (2016). Visually
indicated sounds. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages
2405–2413.
Petridis, S. and Pantic, M. (2016). Deep complementary bottleneck features for visual speech recognition.
In Acoustics, Speech and Signal Processing (ICASSP), 2016 IEEE International Conference on, pages
2304–2308. IEEE.
Rix, A., Beerends, J., Hollier, M., and Hekstra, A. (2001). Perceptual evaluation of speech quality (pesq), an
objective method for end-to-end speech quality assessment of narrowband telephone networks and speech
codecs. ITU-T Recommendation, 862.
Srivastava, N., Hinton, G. E., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. (2014). Dropout: a simple
way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958.
Wand, M., Koutník, J., and Schmidhuber, J. (2016). Lipreading with long short-term memory. In Acoustics,
Speech and Signal Processing (ICASSP), 2016 IEEE International Conference on, pages 6115–6119. IEEE.

9

