Learning to Have an Ear for Face Super-Resolution
Givi Meishvili Simon Jenni Paolo Favaro









Learning to Have an Ear for Face Super-Resolution
    Givi Meishvili   Simon Jenni   Paolo Favaro

University of Bern, Switzerland

{givi.meishvili, simon.jenni, paolo.favaro}@inf.unibe.ch

    December 30, 2023
===========================================================================================================================================

empty




We propose a novel method to use both audio and a low-resolution image to perform extreme face super-resolution (a 16× increase of the input size). When the resolution of the input image is very low (e.g., 8× 8 pixels), the loss of information is so dire that important details of the original identity have been lost and audio can aid the recovery of a plausible high-resolution image. In fact, audio carries information about facial attributes, such as gender and age. 
To combine the aural and visual modalities, we propose a method to first build the latent representations of a face from the lone audio track and then from the lone low-resolution image. 
We then train a network to fuse these two representations. 
We show experimentally that audio can assist in recovering attributes such as the gender, the age and the identity, and thus improve the correctness of the high-resolution image reconstruction process. Our procedure does not make use of human annotation and thus can be easily trained with existing video datasets.
Moreover, we show that our model builds a factorized representation of images and audio as it allows one to mix low-resolution images and audio from different videos and to generate realistic faces with semantically meaningful combinations.








§ INTRODUCTION


Image super-resolution is the task of recovering details of an image that has been captured with a limited resolution. 
Typically, the resolution of the input image is increased by a scaling factor of 4× to 8×.

In the more extreme case, where the scaling factor is 16× or above, the loss of detail can be so considerable that important semantic information is lost. This is the case, for example, of images of faces at an 8× 8 pixels resolution, where information about the original identity of the person is no longer discernible. The information still available in such a low-resolution image is perhaps the viewpoint and colors of the face and the background. While it is possible to hallucinate plausible high-resolution images from such limited information, useful attributes such as the identity or even just the gender or the age might be incorrect (see Fig. <ref> (a)-(d)).

If the low-resolution image of a face is extracted from a video, we could also have access to the audio of that person. Despite the very different nature of aural and visual signals, they both capture some shared attributes of a person and, in particular, her identity. In fact, when we hear the voice of an iconic actor we can often picture his or her face in our mind. 
<cit.> recently showed that such capability can be learned by a machine as well.
The possibility to recover a full identity is typically limited to a set of known people (, celebrities). 
Nonetheless, even when the identity of a person is completely new, her voice indicates important facial attributes such as gender, age, and ethnicity. If such information is not present in the visual data (, with a low-resolution image), audio could be a benefit to image processing and, in particular, image super-resolution (see Fig. <ref> (e)-(f)). 
For example, in videos where the identity of a speaker is hidden via pixelation, as shown in Fig. <ref>, audio could be used to recover a more plausible face than from the lone low-resolution image.




Therefore, we propose to build a model for face super-resolution by exploiting both a low-resolution image and its audio.
To the best of our knowledge, this has never been explored before.
A natural way to solve this task is to build a multimodal network with two encoding networks, one for the low-resolution image and one for audio, and a decoding network mapping the concatenation of the encoders outputs to a high resolution image. 
In theory, a multi-modal network should outperform its uni-modal counterparts. In practice, however, this does not happen with standard networks and training strategies, as shown empirically in <cit.>. 
According to <cit.> the performance gap is due to: 1) the difference between modalities in term of convergence and over-fitting speeds, 2) The susceptibility of multi-modal architectures to over-fitting due to their higher capacity. 
To address the above training issues of multi-modal networks, we propose to train the low-resolution image encoder and the audio encoder separately, so that their disentanglement accuracy can be equalized. 
To this aim, we first train a generator 

that starts from a Gaussian latent space and outputs high resolution images (see Fig. <ref>). The generator is trained as in the recent StyleGAN of <cit.>, which produces very high quality samples and a latent space with a useful hierarchical structure. Then, we train a reference encoder to invert the generator by using an autoencoding constraint. The reference encoder maps a high-resolution image to the latent space of the generator, which then outputs an approximation of the input image. Then, given a matching high/low-resolution image pair, we pre-train a low-resolution image encoder 

to map its input to the same latent representation of the reference encoder (on the high-resolution image). As a second step, we train an audio encoder 

and a fusion network to improve the latent representation of the (fixed) low-resolution image encoder.

To speed up the training of the audio encoder we also pre-train it by using as latent representation the average of the outputs of the reference encoder on a high-resolution image and its horizontally mirrored version. Thanks to the hierarchical structure of the latent space learned through StyleGAN, this averaging removes information, such as the viewpoint, that  audio cannot possibly carry. In Section <ref>, we describe in detail the training of each of the above models. Finally, in Section <ref> we demonstrate experimentally that the proposed architecture and training procedure successfully fuses aural and visual data. We show that the fusion yields high resolution images with more accurate identities, gender and age attributes than the reconstruction based on the lone low-resolution image. We also show that the fusion is semantically meaningful by mixing low-resolution images and audio from different videos 
(see an example in Fig. <ref> (b)).

Contributions:
Our method builds three models for the following mappings: 1) Audio to high-resolution image; 2) Low-resolution image to high-resolution image; 3) Audio and low-resolution image to high-resolution image. The first mapping was developed concurrently to Speech2Face <cit.>. A notable difference is that Speech2Face is trained using as additional supervision a pre-trained face recognition network, while our method is fully unsupervised. In the second mapping, we show in our Experiments section that we achieve state of the art performance at 16×. In the last mapping, which is the main novelty of this paper, we show that our trained model is able to transfer and combine facial attributes from audio and low-resolution images.[Our code and pre-trained models are available at https://gmeishvili.github.io/ear_for_face_super_resolution/index.htmlhttps://gmeishvili.github.io/ear_for_face_super_resolution/index.html.]







§ RELATED WORK

General Super-Resolution. 
Singe Image Super Resolution (SISR) is a very active research area, which largely benefitted from the latest developments in deep learning (see, , <cit.>).
A wide set of instances of this problem has been addressed, ranging from arbitrary scale factors <cit.>, to improving the realism of the training set through accurate modeling <cit.> or through using real zoomed-in images <cit.>, to robustness against adversarial attacks <cit.> and generalization <cit.>, and to modeling multiple degradations <cit.>. Finally, 
<cit.> focus on the evaluation of the image quality in SISR.
Advances in general super-resolution have also been largely driven by the introduction of task-specific network architectures and components (see , <cit.>).
In our method, we do not rely on task-specific architectures, although we leverage the design of a state-of-the-art generative model <cit.>. 

Face Super-Resolution. 
The face super-resolution problem has been tackled with a wide variety of approaches. For example, Huang  <cit.> trained a CNN to regress wavelet coefficients of HR faces and Yu  <cit.> introduced a transformative discriminative autoencoder to super-resolve unaligned and noisy LR face images. 
More in general, recent methods addressed the problem by using additional supervision, for example, in the form of facial landmarks, heat-maps or the identity label, and multi-task learning (, <cit.>). 
In contrast, by using videos with corresponding audio tracks our method does not rely on additional human annotation and thus its training can scale more easily to large datasets.

GAN-Based Super-Resolution. 
Many general super-resolution methods also make use of adversarial training (see, , <cit.>).
Several super-resolution methods based on Generative Adversarial Networks (GANs) <cit.> focus specifically on faces <cit.>. 
Our work also relies on the use of a GAN to learn a face specific prior. 
However, our approach builds a more general generative network that combines low-resolution images and audio (see Fig. <ref>).

Use of Audio in Vision Tasks. 
The use of audio in combination with video has received a lot of attention recently (see, , <cit.>).
Audio and video have been combined to learn to localize objects or events <cit.>, to learn how to separate audio sources <cit.>, to learn the association between sound and object geometry and materials <cit.>, and to predict body dynamics <cit.>.
A significant body of work has also been devoted to the mapping of audio to visual information (see, , <cit.> and references therein).
However, to the best of our knowledge we are the first to combine audio and images for an image restoration task.




§ EXTREME FACE SUPER-RESOLUTION WITH AUDIO


Our goal is to design a model that is able to generate high resolution images based on a (very) low resolution input image and an additional audio signal. 
The dataset is therefore given by 𝒟 = { (x_i^h, x_i^l, a_i)| i=1, …, n } where x_i^h is the high-resolution image, x_i^l is the low-resolution image and a_i is a corresponding audio signal. 
Our model consists of several components: a low-resolution encoder E_l, an audio encoder E_a, a fusion network F and a face generator G. 
An overview of the complete architecture is given in Fig. <ref>.


 §.§ Combining Aural and Visual Signals

As mentioned in the Introduction, a natural choice to solve our task is to train a feedforward network to match the ground truth high resolution image given its low-resolution image and audio signal.
Experimentally, we found that such a system tends to ignore the audio signal and to yield a one-to-one mapping from a low-resolution to a single high-resolution image. 
We believe that this problem is due to the different nature of the aural and visual signals, and the choice of the structure of the latent space.
The fusion of both signals requires mapping their information to a common latent space through the encoders.
However, we find experimentally that the audio signal requires longer processing and more network capacity to fit the latent space (this is also observed in <cit.>).
This fitting can also be aggravated by the structure of the latent space, which might be biased more towards images than audio.
Ideally, the low-resolution image should only condition the feedforward network to produce the most likely corresponding high-resolution output and the audio signal should 
introduce some local variation (, modifying the gender or the age of the output). 
Therefore, for the fusion to be effective it would be useful if the audio could act on some fixed intermediate representation from the low-resolution image, where face attributes present in the audio are disentangled.

For these reasons we opted to pre-train and fix the generator of a StyleGAN <cit.>
and then to train encoders to autoencode the inputs by using the generator as a decoder network.
StyleGAN generators have been shown to produce realistic high resolution images along with a good 
disentanglement of some meaningful factors of variation in the intermediate representations. 
Such models should therefore act as good priors for generating high resolution face images and 
the disentangled intermediate representations should allow better editing based on the audio signal. 
Formally, we learn a generative model of face images G(z), where z ∼𝒩(0, I_d), 
by optimizing the default non-saturating loss of StyleGAN (see <cit.> for details).


 §.§ Inverting the Generator


Our goal is that the fusion of the information provided by the low-resolution image and audio track results in a reconstruction that is close to the corresponding high resolution image. We pose this task as that of mapping an image x to its latent space target z, such that G(z) = x. In other words, we need to invert the pre-trained generator G.
Recently, this problem has attracted the attention of the research community <cit.>. 
In this paper, we introduce a novel GAN inversion approach, where we first pre-train the encoder E_h while the generator is fixed, and then we train both the encoder E_h and the generator G (fine-tuning) through an autoencoding constraint and by anchoring the weights of G to its initial values through an L_2 loss. Then, the latent representation z_i corresponding to the image x_i can be generated by the encoder E_h, and used as a target by the encoders of the low-resolution images and the audio, and the fusion network.

Encoder Pre-Training. As a first step we train a high-resolution image encoder by minimizing the loss

    L_pre-train = ∑_i=1^n |G( z_i)-x_i^h|_1 + λ_fℓ_feat(G( z_i ), x_i^h),

only with respect to E_h, where z_i = E_h(x_i^h), ℓ_feat is a perceptual loss based on VGG features (see Supplementary material for more details), and λ_f=1 is a coefficient that regulates the importance of ℓ_feat relative to the L_1 loss.
We found that regressing a single z_i is not sufficient to recover a good approximation of x_i^h. 



In the original style-based generator <cit.> each z_i is mapped to a vector w_i, which is then replicated and inserted at k different layers of the generator (each corresponding to different image scales). 
To improve the high-resolution reconstruction, we instead generate k different z_ij, j=1,…,k, and feed the resulting w_ij to the corresponding layers in the generator. The output of E_h therefore lies in ℝ^k × d. 
Note that this is not too dissimilar from the training of the style-based generator, where the w-s of different images are randomly mixed at different scales.



Encoder and Generator Fine-Tuning.
This second optimization problem can be written as

    min _E_h,G    L_pre-train
    	
    	+λ_t|G_init - G|_2^2,

where λ_t=1 is a coefficient that regulates how much G can be updated, and G_init denotes the weights of G after StyleGAN training.
Moreover, during training we relax the regularizer of the weights of G by reducing λ_t by a factor of 2 as soon as the overall loss is minimized (locally). The purpose of the pre-training and the regularizer decay procedure is to encourage a gradual convergence of both the encoder and the decoder without losing prematurely the structure of the latent representation of G. 
Examples of inversions before and after the fine-tuning are shown in Fig. <ref>. There is a visible improvement in the reconstruction accuracy of both the face and the background. Quantitative results are shown in the Experiments section.



 §.§ Pre-Training Low-Res and Audio Encoders

Given the high-resolution image encoder, we now have targets z_i for the low-resolution and audio fusion. 
However, the training of a fusion model directly on these targets runs into some difficulties.
As mentioned before, we find experimentally that, given enough capacity, a fusion model F(x_i^l, a_i) trained to predict z_i = E_h(x_i^h), ignores the audio signal a_i almost completely. 
To address this degenerate behavior, we train two encoders E_l and E_a separately to extract as much information from the two modalities as possible and only later fuse them.
To ensure that neither of the two encoders can overfit the whole training set 𝒟 we extract the subset 𝒟_pre = { (x_i^h, x_i^l, a_i)| i=1, …, n/2 } for the encoders pre-training and use the entire 𝒟 only for the later fusion training.
The low-resolution encoder E_l is trained to regress the high-resolution encodings z_i=E_h(x_i^h) from x_i^l by solving

    min _E_l∑_x_i^l, x_i^h ∈𝒟_pre|E_l(x_i^l) - z_i|_1+ λ|D∘ G( E_l(x_i^l)) - x_i^l|_1,

where D∘ x is the 16× downsampling of x and λ=40.

In the case of the audio encoding, regressing all the information in z_i with E_a(a_i) is not possible without overfitting, as many of the factors of variation in z_i, , the pose of the face, are not present in a_i. 
To remove the pose from z_i we generate the targets for the audio encoder as z̅_̅i̅≐1/2( E_h(x_i^h)+E_h(x̂_i^h)), where x̂_i^h is a horizontally flipped version of the image x_i^h. As it turns out, due to the disentangled representations of G, the reconstruction G(z̅_̅i̅) produces a neutral frontal facing version of G(z_i) (see Fig. <ref>). The audio encoder E_a is finally trained by solving

    min _E_a∑_a_i, x_i^h ∈𝒟_pre|E_a(a_i) - z̅_̅i̅|_1 .



 §.§ Fusing Audio and Low-Resolution Encodings

We now want to fuse the  information provided by the pre-trained encoders E_l and E_a. 
Since the low-resolution encoder E_l already provides a good approximation to E_h, it is reasonable to use it as a starting point for the final prediction. 
Conceptually, we can think of E_l as providing a z_i^l=E_l(x_i^l) that results in a canonical face G(z_i^l) corresponding to the low-resolution image x_i^l.
Ambiguities in z_i^l could then possibly be resolved via the use of audio, which would provide an estimate of the residual Δ z_i = z_i-z_i^l.
We therefore model the fusion mechanism as z_i^f=E_l(x_i^l)+F(E_l(x_i^l), E_a(a_i)), where F is a simple fully-connected network acting on the concatenation of E_l(x_i^l) and E_a(a_i). 
Since the audio-encoding E_a might be suboptimal for the fusion, we continue training it along with F. 
The limited complexity of the function F prevents the overfitting to the low-resolution encoding, but provides the necessary context for the computation of Δ z_i. To summarize, we train the fusion by optimizing 

    min _E_a, F∑_a_i, x_i^h, x_i^l ∈𝒟|z_i^f - z_i|_1
    	+ λ|D∘  G( z_i^f) - x_i^l|_1.















§ EXPERIMENTS


We demonstrate our contributions by evaluating three models with different input-output mappings: 
1) Audio to high-resolution image; 
2) Low-resolution image to high-resolution image; 
3) Audio and low-resolution image to high-resolution image.
In particular, we focus our attention on the third case as it is the main objective of this paper.



 §.§ Dataset

We perform all our experiments on a subset of the VoxCeleb2 dataset <cit.>. 
The dataset contains over one million audio tracks extracted from 145K videos of people speaking. 

For the full training set 𝒟 we select 104K videos with 545K audio tracks and extract around 2M frames at 128 × 128 pixels such that each speaker has at least 500 associated frames. 
We then extract half of this dataset to create 𝒟_pre in such a way that 𝒟_pre and 𝒟 contain the same speakers, but 𝒟_pre has fewer videos than 𝒟. 
For the test set, we select 39K frames and 37K utterances from 25K videos not contained in the training set (again from the same speakers).
In the end, we select around 4K speakers out of the 6K speakers in the full dataset (filtering out speakers with very few videos and audio tracks).
Note that this selection is purely done to allow the evaluation via a speaker identity classifier.  
We call experiments closed set when the training and test sets share the same set of face identities; instead, we call them open set when the test set has identities that were not in the training set.


 §.§ Implementation

The style-based generator G is pre-trained on the full training set 𝒟 with all hyper-parameters set to their default values (see <cit.> for details). 
It has been trained on a total of 31 million images. 
The high-resolution encoder E_h is trained for 715K iterations and a batch-size of 128 on the 128 × 128 images from 𝒟.
The low-resolution encoder E_l and the audio encoder E_a are trained on 𝒟_pre. 
E_l is trained for 240K iterations with a batch-size of 256 and E_a is trained for 200K iterations and a batch-size of 64. 
The inputs x_i^l to E_l are of size 8 × 8 pixels and the inputs to E_a are the audio log-spectrograms of a_i of size 257 × 257.  
The fine-tuning of E_a and the training of the fusion layer F is performed for 420K iterations on  𝒟.
We use the Adam optimizer <cit.> with a fixed learning rate of 10^-4 for the training of all the networks. 
A detailed description of the network architectures can be found in the supplementary material. During the training of F we sample one frame and an audio segment (4 seconds) independently (, they are not synchronized) from the same short video. This forces the network to learn more general attributes such as gender and age, rather than characteristics specific to a particular instant in time.



 §.§ Audio-Only to High Resolution Face


Although our main objective is to obtain super-resolved images from the fusion of low-resolution images and audio, we provide a brief comparison between our model for face reconstruction from audio (E_a+G) with Speech2Face <cit.>. Since the dataset of <cit.> is not public, we perform a qualitative and a quantitative comparison based on audio tracks and reconstrucitons by Oh  <cit.> from <https://speech2face.github.io/supplemental/retrieve/index.html>. In Fig. <ref> we show the reference faces obtained by Speech2Face and our output using the same audio tracks. We can see that the gender and age are matching. In the second evaluation, we perform gender classification on the output of our audio-to-image model when given audio from the VoxCeleb dataset <cit.> as input. Given a voice of the male or female person, our E_a+G model generates faces of males and females in 97% and 96% of the cases respectively. The results match those reported by <cit.>. Notice that <cit.> uses supervision from a classifier during training, while our training is completely unsupervised. 
 
 
 


 §.§ 
Classification 
as a Performance Measure

To evaluate the capability of our model to recover gender and other identity attributes based on the low-resolution and audio inputs we propose to use the accuracy of a pre-trained identity classifier C_i and gender classifier C_g which achieve an accuracy of 95.25% and 99.53% respectively on the original high resolution images.
To this end, we fine-tune two VGG-Face CNNs of <cit.> on the training set 𝒟 for 10 epochs on both face attributes. 
As one can see in Table <ref> these classifiers perform well on the test set on both face attributes. Although we do not have the ground truth age of our dataset, we use a pre-trained age classifier C_a <cit.> as the reference. Then, we measure the performance of our models by checking the consistency between the classified age of the input and of the output.



Ablations. We perform ablation experiments to understand the information retained in the encoders and to justify the design of our final model. The accuracy of the classifiers C_i and C_g, as well as the consistency error of C_a, are reported in Table <ref> for the following ablation experiments


  (a)-(b) The importance of fine-tuning. In (a) we show the performance after pre-training of E_h without fine-tuning, and in (b) we show the improvement in performance with the fine-tuning of G as in eq. (<ref>).

  (c)-(d) Individual components. Shows the performance of the individual encoders without fusion. Results for the low-resolution encoder E_l and the audio encoder E_a should be compared to the reference high-resolution encoder E_h. 

  (e)-(h) Fusion strategies. The performance of different fusion strategies is reported. As a reference, we report results of a fusion model F_1 with a single fully-connected layer and fine-tuning of E_a. We compare it to a more complex fusion network F with three fully-connected layers when the audio is not used (f), the audio encoder is fixed (g) and when fine-tuning E_a (h).

Ablation (c) and (d) show that E_a is able to predict the correct gender more often than E_l. 
All the fusion approaches (e)-(h) lead to an improvement in terms of identity prediction over E_a and E_l alone, thus showing that the information from both inputs is successfully integrated. We can observe that both gender and age can be predicted well even from unseen identities (, the open-set case). Ablation (f) vs (h) shows that the method is indeed using information from the audio signal and the performance increase is not due to the additional capacity of the fusion network F. Ablation (h) vs (e) justifies the usage of 3 fully-connected layers instead of only 1. Ablation (h) vs (g) demonstrates that fine-tuning of the encoder E_a during the training of the fusion network F leads to slight improvements in terms of our quantitative metrics. Note that the performance of all methods in Table <ref>, including the SotA <cit.>, is lower in the open set experiments than in the closed set ones. This is expected since all methods are trained only on identities present in the training set and most likely only a small amount of information is shared across identities. The open set experiments show how much the methods can identify such shared information, which is a sign of generalization.
See also Fig. <ref> for qualitative results.



Comparisons to Other Super-Resolution Methods.
We compare to state-of-the-art super-resolution methods in Table <ref> and Fig. <ref>. 
The standard metrics PSNR and SSIM along with the accuracy of C_i and C_g, and the errors of C_a are reported for super-resolved images of our test set. 
Note that most methods in the literature are not trained on extreme super-resolution factors of 16×, but rather on factors of 4×. Therefore, we report the results of one method using a factor of 4× as a reference for the changes with the 16× factor. The performance of other 4× super-resolution methods can be found in our supplementary material.
We retrain the methods of <cit.> and <cit.> on our training set, before evaluating their performance. Notice that although LapSRN trained on 16× super-resolution performs better in terms of PSNR and SSIM than our method, the quality of the recovered image is clearly worse (see Fig. <ref>). This difference in the quality is instead revealed by evaluating the gender and identity classification accuracies, and the age classification error of the restored images. This suggests that while PSNR and SSIM may be suitable metrics to evaluate reconstructions with small super-resolution factors, they may not be suitable to assess the reconstructions in more extreme cases such as with a factor of 16×.



Editing by Mixing Audio Sources. 
Our model allows us to influence the high-resolution output by interchanging the audio tracks used in the fusion.
To demonstrate this capability we show examples where we mix a fixed low-resolution input with several different audio sources in Fig. <ref>. 

To also quantitatively evaluate such mixing, we feed low-resolution images and audios from persons of different gender and classify the gender of the resulting high-resolution faces. In Table <ref>, we report the accuracy with respect to the ground-truth gender labels of low-resolution images and audios.


Failure Cases. We observe that failures may correspond more to the inherent bias presented in the training set than the training algorithm or network architecture. Failure cases sometimes happen when the gender can be easily guessed just from the low-resolution image. Some of the failure cases are reported in Fig <ref>.



§ CONCLUSIONS

We have introduced a new paradigm for face super-resolution, where also audio contributes to the restoration of missing details in the low-resolution input image. We have described the design of a neural network and the corresponding training procedure to successfully make use of the audio signal despite the difficulty of extracting visual information from it. We have also shown quantitatively that audio can contribute to improving the accuracy of the identity as well as the gender and age of the restored face. Moreover, we have shown that it is possible to mix low-resolution images and audios from different videos and obtain semantically meaningful high resolution images. A fundamental challenge in this work was the fusion of information from these very different modalities. As we have shown, valuable information is present in both. However, we observed that a naïve end-to-end training tends to ignore audio information. We conjecture that this problem might be a fundamental issue with current training schemes of neural networks and its solution could provide insights on how to improve the training on tasks in general. 


Acknowledgements. This work was supported by grants 165845&169622 of the Swiss National Science Foundation.

ieee_fullname



