Title: Improved Speech Reconstruction from Silent Video,
Abstract: Speechreading is the task of inferring phonetic information from visually
observed articulatory facial movements, and is a notoriously difficult task for
humans to perform. In this paper we present an end-to-end model based on a
convolutional neural network (CNN) for generating an intelligible and
natural-sounding acoustic speech signal from silent video frames of a speaking
person. We train our model on speakers from the GRID and TCD-TIMIT datasets,
and evaluate the quality and intelligibility of reconstructed speech using
common objective measurements. We show that speech predictions from the
proposed model attain scores which indicate significantly improved quality over
existing models. In addition, we show promising results towards reconstructing
speech from an unconstrained dictionary.