Title: Audio-Visual Speech Enhancement based on Multimodal Deep Convolutional Neural Network,
Abstract: Speech enhancement (SE) aims to reduce noise in speech signals. Most SE
techniques focus on addressing audio information only. In this work, inspired
by multimodal learning, which utilizes data from different modalities, and the
recent success of convolutional neural networks (CNNs) in SE, we propose an
audio-visual deep CNN (AVDCNN) SE model, which incorporates audio and visual
streams into a unified network model. In the proposed AVDCNN SE model, audio
and visual data are first processed using individual CNNs, and then, fused into
a joint network to generate enhanced speech at the output layer. The AVDCNN
model is trained in an end-to-end manner, and parameters are jointly learned
through back-propagation. We evaluate enhanced speech using five objective
criteria. Results show that the AVDCNN yields notably better performance,
compared with an audio-only CNN-based SE model and two conventional SE
approaches, confirming the effectiveness of integrating visual information into
the SE process.