Generative Collaborative Networks for Single Image Super-Resolution
Mohamed El Amine Seddik Mohamed Tamaazousti John Lin













mymainaddress,mysecondaryaddress]Mohamed El Amine Seddik
mohamedelamine.seddik@cea.fr

mymainaddress]Mohamed Tamaazousti

mohamed.tamaazousti@cea.fr

mymainaddress,johnaddress]John Lin
john.lin@cea.fr

[mymainaddress]CEA/LIST/DIASI/LVIC, F-91191 Gif-sur-Yvette, France
[mysecondaryaddress]CentraleSupelec/L2S, 3 rue Joliot Curie, 91192 Gif-sur-Yvette, France
[johnaddress]ISIT-UMR6284 CNRS/Auvergne University, Clermont-Ferrand, France


A common issue of deep neural networks-based methods for the problem of Single Image Super-Resolution (SISR), is the recovery of finer texture details when super-resolving at large upscaling factors. This issue is particularly related to the choice of the objective loss function. In particular, recent works proposed the use of a VGG loss which consists in minimizing the error between the generated high resolution images and ground-truth in the feature space of a Convolutional Neural Network (VGG19), pre-trained on the very “large” ImageNet dataset. When considering the problem of super-resolving images with a distribution “far” from the ImageNet images distribution (e.g., satellite images), their proposed fixed VGG loss is no longer relevant. In this paper, we present a general framework named Generative Collaborative Networks (GCN), where the idea consists in optimizing the generator (the mapping of interest) in the feature space of a features extractor network. The two networks (generator and extractor) are collaborative in the sense that the latter “helps” the former, by constructing discriminative and relevant features (not necessarily fixed and possibly learned mutually with the generator). We evaluate the GCN framework in the context of SISR, and we show that it results in a method that is adapted to super-resolution domains that are “far” from the ImageNet domain.



Super-ResolutionDeep LearningGANs Perceptual Loss











§ INTRODUCTION

The super-resolution problem (𝒫_sr) consists in estimating a high resolution (HR) image from its corresponding low resolution (LR) counterpart.
𝒫_sr finds a wide range of applications and has attracted much attention within the community of computer vision <cit.>. Generally, the considered optimization objective of supervised methods to solve 𝒫_sr is the minimization of the mean squared error (MSE) between the recovered HR image and ground-truth. This class of methods are known to be suboptimal to reconstruct texture details at large upscaling factors. In fact, since MSE consists in a pixel-wise images differences, its ability to recover high texture details is limited <cit.>. Furthermore, the minimization of MSE maximizes the Peak Signal-to-Noise-Ratio (PSNR) metric, which is commonly used for the evaluation of 𝒫_sr methods <cit.>.In order to correctly recover finer texture details when super-resolving at large upscaling factors, a recent (state-of-the-art) work <cit.> defined a perceptual loss which is a combination of an adversarial loss and a VGG loss. The former encourages solutions perceptually hard to distinguish from the HR ground-truth images, while the latter consists in using high-level feature maps of the VGG network <cit.> pre-trained on ImageNet <cit.>. When considering the problem of super-resolving images from a target-domain different than ImageNet (e.g., satellite images), the features produced by the pre-trained VGG network on the source domain (ImageNet) are suboptimal and no longer relevant for the target domain. In fact, transfer-learning methods are known to be efficient only when the source and target domains are close enough <cit.>. In this work, we present a general framework which we call Generative Collaborative Networks (GCN), where the main idea consists in optimizing the generator (i.e., the mapping of interest) in the feature space of a network which we shall refer to as a features extractor network. The two networks are said to be collaborative in the sense that the features extractor network “helps” the generator by constructing (here, learning) relevant features. In particular, we applied our framework to the problem of single image super-resolution, and we demonstrated that it results in a method that is more adapted (compared to SRGAN <cit.>) when super-resolving images from a domain that is “far” from the ImageNet domain.The rest of the paper is organized as follows. In Section <ref> we present the state of the art on the problem of single image super-resolution. We describe our Generative Collaborative Networks framework in Section <ref>. Section <ref> presents our proposed method for the super resolution task and related experimental results. Section <ref> provides some discussions and concludes the article.




§ RELATED WORK
 

The problem of super-resolution has been tackled with a large range of approaches. In the following, we will consider the problem of single image super-resolution (𝒫_sisr) and thus the approaches that recover HR images from multiple images <cit.> are out of the scope of this paper. First approaches to solve 𝒫_sisr were filtering-based methods (e.g., linear, bicubic or Lanczos <cit.> filtering). Even if these methods are generally very fast, they usually yield overly smooth textures solutions <cit.>. Most promising and powerful approaches are learning-based methods which consist in establishing a mapping between LR images and their HR counterparts (supposed to be known). Initial work was proposed by Freeman et al. <cit.>. This method has been improved in <cit.> by using compressed sensing approaches. Patch-based methods combined with machine learning algorithms were also proposed: in <cit.> upsampling a LR image by finding similar LR training patches in a low dimensional space (using neighborhood embedding approaches) and a combination of the HR patches counterparts are used to reconstruct HR patches. A more general mapping of example pairs (using kernel ridge regression) was formulated by Kim and Kwon <cit.>. Similar approaches used Gaussian process regression <cit.>, trees <cit.> or Random Forests <cit.> to solve the regression problem introduced in <cit.>. An ensemble method-based approach was adopted in <cit.> by learning multiple patch regressors and selecting the most relevant ones during the test phase.Convolutional neural networks (CNN)-based approaches outperformed other 𝒫_sisr approaches, by showing excellent performance. Authors in <cit.> used an encoded sparse representation as a prior in a feed-forward CNN, based on the learned iterative shrinkage and thresholding algorithm   of <cit.>. An end-to-end trained three layer deep fully convolutional network, based on bicubic interpolation to upscale the input images, was used in <cit.> and achieved good 𝒫_sisr performances. Further works suggested that enabling the network to directly learn the upscaling filters, can remarkably increase performance in terms of both time complexity and accuracy <cit.>. In order to recover visually more convincing HR images, Johnson et al. <cit.> and Bruna et al. <cit.> used a closer loss function to perceptual similarity. More recently, authors in <cit.> defined a perceptual loss which is a combination of an adversarial loss and a VGG loss. The latter consists in minimizing the error between the recovered HR image and ground-truth in the high-level feature space of the pre-trained VGG network <cit.> on ImageNet <cit.>. This method notably outperformed CNN-based methods for the problem 𝒫_sisr.




§ GENERATIVE COLLABORATIVE NETWORKS




 §.§ Proposed Framework

Consider a problem 𝒫 of learning a mapping function ℱ, parameterized by θ_ℱ, that transforms images from a domain 𝒳 to a domain 𝒴, given a training set of N pairs { (x_i,y_i)}_i=1^N∈𝒳×𝒴. Denote by p_𝒳 and p_𝒴 the probability distributions respectively over 𝒳 and 𝒴. In addition, we introduce a given features extractor function denoted Φ, parameterized by θ_Φ, that maps an image y∈𝒴 to a certain euclidean feature space 𝒮_Φ of dimensionality d. The mappings ℱ and Φ are typically feed-forward Convolutional Neural Networks. The Generative Collaborative Networks (GCN) framework consists in learning the mapping function ℱ by minimizing a given loss function[ℓ_2-loss is considered in the following.] in the space of features 𝒮_Φ, between the generated images (through ℱ) and ground-truth. Formally,

    θ̂_ℱ = _θ_ℱλ_1/N  d∑_i=1^N ∑_j=1^d ( Φ_j ( y_i ) - Φ_j ( ℱ(x_i) )  )^2 + λ_2 Ω(θ_ℱ),

where Ω(θ_ℱ) is a certain regularization term (detailed below) on the weights θ_ℱ and λ_1 and λ_2 are summation coefficients. The two networks ℱ and Φ are collaborative in the sense that, the latter learns specific features of the domain 𝒴 and “helps” the former, as it is learned in the space 𝒮_Φ. An important question arises about how to learn the mapping Φ. In following, we describe different classes of methods depending on the learning strategy of Φ. In fact, the features extractor function Φ can take different forms and be learned by different strategies. In particular, we distinguish two learning strategies (illustrated in Figure <ref>), which we shall call disjoint-learning and joint-learning. The four following cases belong to the disjoint-learning strategy:



  (1.a) When Φ is the identity operator (Φ =). In that case, the objective in Eq.(<ref>) becomes a simple pixel-wise MSE loss function. We refer to this class of methods by 𝒫/mse.

  (1.b) When Φ corresponds to a random feature map neural network, that is to say, the weights θ_Φ are set randomly according to a given distribution μ. We refer to this class of methods by 𝒫/ran.

  (1.c) When Φ is a part of a model that solves a reconstruction problem (jointly with an auxiliary mapping function Ψ:𝒮_Φ→𝒴), by minimizing the pixel-wise ℓ_2-loss function between the reconstructed images (through Ψ) and ground-truth:


    (θ̂_Φ,_)= _(θ_Φ, θ_Ψ)1/N (𝒴)∑_i=1^N ∑_j=1^(𝒴)( (y_i)_j - (Ψ∘Φ (y_i))_j)^2.

Notably, this strategy allows for the learning of reconstruction features which are different from classification-based features. We refer to this class of methods by 𝒫/rec.

  (1.d) When Φ is trained to solve a multi-label classification problem <cit.>, that is to say, when labels are available for the domain 𝒴. More precisely, it exists a dataset { (y_i, c_i) }_i=1^n∈𝒴×{1,…,m} of n images labelled among m classes and Φ is learned to minimize the following objective:

    (θ̂_Φ,_)= _(θ_Φ, θ_Ψ)ℙ{Ψ∘Φ(y_i) =c_i  |  y_i  ;  i∈{1,…,m}},
 
where Ψ:𝒮_Φ→{1,…,m}. We refer to this class of methods by 𝒫/cla.

The features extractor function Φ can also be trained jointly with the desired mapping function ℱ. Indeed, as in the GANs paradigm, one can use a discriminator to distinguish the generated images (through ℱ) and ground-truth, and thus learn more relevant and specific features for the problem of interest 𝒫. In particular, the joint-learning strategy contains two cases:


  (2.a) When Φ is a part of a discriminator. 𝒟 = Ψ∘Φ:𝒴→{0,1} that classifies the generated images (through ℱ) and ground-truth. 𝒟 is optimized in an alternating manner along with ℱ to solve the adversarial min-max problem <cit.>:

    min_θ_ℱmax_  (θ_Φ, θ_Ψ)  𝔼_y∼ p_𝒴[ logΨ∘Φ (y) ] + 𝔼_x∼ p_𝒳[ log{  1 - Ψ∘Φ∘ℱ(x) }].

The adversarial loss (second term of Eq. (<ref>)) can thus be seen as a regularization of the parameters θ_ℱ by affecting this quantity to Ω(θ_ℱ) in Eq. (<ref>). This regularization “pushes” the solution of the problem in Eq. (<ref>) to the manifold of the images in the domain 𝒴. We refer to this class of methods by 𝒫/adv. When λ_2=0, we refer to it by 𝒫/dis.

  (2.b) When Φ is a part of a discriminator and an auto-encoder. Namely, by optimizing its weights θ_Φ to solve simultaneously, an adversarial problem as in Eq. (<ref>); through 𝒟 = Ψ_1 ∘Φ:𝒴→{0,1}, and a reconstruction problem as in Eq. (<ref>); through a mapping Ψ_2:𝒮_Φ→𝒴. We refer to this class of methods by 𝒫/adv,rec or 𝒫/dis,rec depending on the value of λ_2 in Eq. (<ref>).




 §.§ Existing Loss Functions

The natural way to learn a mapping from a manifold to another is to use 𝒫/mse methods. It is well known <cit.> that this class of methods lead to overly-smooth and poor perceptual quality solutions. In order to handle the mentioned perceptual quality limitation, a variety of methods have been proposed in the literature. First methods used generative adversarial networks (GANs) for generating high perceptual quality images <cit.>, style transfer <cit.> and inpainting <cit.>, namely the class of methods 𝒫/adv with λ_1=0. Authors in <cit.> proposed to use 𝒫/mse with an adversarial loss (λ_1>0 and λ_2>0) to train a network that super-resolves face images with large upscaling factors. Authors in <cit.> and in <cit.> used 𝒫/cla by considering respectively Φ=VGG19 and Φ=AlexNet networks as fixed features extractors (learned disjointly from the mapping of interest), which result in a more perceptually convincing results for both super-resolution and artistic style-transfer <cit.>. More recently, authors in <cit.> used 𝒫/cla,adv by considering Φ=VGG19 as a fixed features extractor combined with an adversarial loss (λ_2>0). To the best of our knowledge, as summarized in table <ref>, the use of the other learning strategies of Φ; namely (1.c), (2.a) and (2.b), have not been explored in the literature. We particularly apply these strategies in the context of Single Image Super-Resolution, which results in methods that are more suitable (comparing to the SRGAN method <cit.>) to super-resolution domains that differ from the ImageNet domain. The proposed methods as well as the corresponding experiments are presented in the following section.  






§ APPLICATION OF GCN TO SINGLE IMAGE SUPER-RESOLUTION




 §.§ Proposed Methods

In this section, we consider the problem of Single Image Super-Resolution (𝒫_sisr). In particular, we suppose we are given N pairs { (I_i^LR,I_i^HR) }_i=1^N of low-resolution images and their high-resolution counterparts. Recalling our GCN framework (presented in Section <ref>) the proposed methods for the problem 𝒫_sisr are: 𝒫_sisr/rec, 𝒫_sisr/dis, 𝒫_sisr/dis,rec, 𝒫_sisr/adv and 𝒫_sisr/adv,rec. We show in the following that the most convincing results are given by 𝒫_sisr/adv,rec. In particular, we show on a dataset of satellite images (different from the ImageNet domain) that our method 𝒫_sisr/adv,rec outperforms the SRGAN method <cit.> by a large margin on the considered domain. Note that, as our goal is to show the irrelevance of the VGG loss for some visual domains (different from ImageNet), we do not consider the well-known SR benchmarks (e.g., Set5, Set14, B100, Urban100) for the evaluation, as these benchmarks are relatively close to the ImageNet domain. The evaluation of the different methods is based on perceptual metrics <cit.> which we recall in the following section.



 §.§ Evaluation Metrics

The evaluation of super-resolution methods (more generally image regression-based methods) requires comparing visual patterns which remains an open problem in computer vision. In fact, classical metrics such as L2/PSNR, SSIM and FSIM often disagree with human judgments (e.g., blurring causes large perceptual change but small L2 change). Thus, the definition of a perceptual metric which agrees with humans perception is an important aspect for the evaluation of 𝒫_sisr methods. Zhang et al. <cit.> recently evaluated deep features across different architectures (Squeeze <cit.>, AlexNet<cit.> and VGG<cit.>) and tasks (supervised, self-supervised and unsupervised networks) and compared the resulting metrics with traditional ones. They found that deep features outperform all classical metrics (e.g., L2/PSNR, SSIM and FSIM) by large margins on their introduced dataset. As a consequence, deep networks seem to provide an embedding of images which agrees surprisingly well with humans judgments.Zhang et al. <cit.> compute the distance between two images x,y with a network[The considered networks are Squeeze<cit.>, AlexNet<cit.> and VGG<cit.> and their "perceptual calibrated" versions which we refer to respectively as Squeeze-l, AlexNet-l and VGG-l. See <cit.> and the provided github project within for further details.] Φ in the following way: 

    d_Φ(x,y) = ∑_l 1/H_l W_l∑_h,w‖ w_l ⊙ (Φ^l(x)_hw - Φ^l(y)_hw ) ‖_2^2,

where Φ^l(·) are the extracted features from layer l and unit-normalized in the channel dimension. w_l is a re-scaling vector of the activations channel-wise at layer l. H_l and W_l are respectively the height and width of the l^th feature map.Thus, we compute the perceptual error (PE) of a 𝒫_sisr method (a mapping ℱ) on a given test-set of N low-resolution images and their high-resolution counterparts Π={ (I_i^LR,I_i^HR) }_i=1^N as the mean distances between the generated images (through ℱ) and ground-truth as follows:

    PE_Φ(Π) = 1/N∑_i=1^N d_Φ (ℱ(I_i^LR), I_i^HR).

Note that we use the implementation of <cit.> to compute the perceptual distances d_Φ(·,·) using six variants which are based on the networks Squeeze<cit.>, AlexNet<cit.> and VGG<cit.> and their “perceptual calibrated” versions. The best method is considered to be the one which minimizes the maximum amount of PEs across different networks Φ∈{Squ, Squ-l, Alex, Alex-l, VGG, VGG-l}.




 §.§ Experiments

The overall goal of this section is to validate our statement about the relevance of the VGG loss when super-resolving images from a different domain than the ImageNet domain. To highlight this aspect, we first present the considered datasets, architectures and training details. Then we select the more appropriate method (across the GCN framework methods) for the 𝒫_sisr problem based on perceptual metrics <cit.>. Finally, we compare our proposed method to some baselines and the state-of-the-art SRGAN method <cit.>, on three different datasets (detailed in the following section). We show in particular that our method outperforms SRGAN on the satellite images domain.


  §.§.§ Datasets



	The idea of replacing the MSE pixel-wise content loss on the image by a loss function that is closer to perceptual similarity is not new. Indeed, <cit.> defined a VGG loss on the feature map obtained by a specific layer of the pre-trained VGG19 network and shows that it fixes the inherent problem of overly smooth results which comes with the pixel-wise loss. Nevertheless, VGG19 being trained on ImageNet, their method would not perform particularly well on different images, the distribution of which is far away from that of ImageNet.
Therefore, we propose a similar method where the difference is that our features extractor is not pre-trained, but trained jointly with the generator. This removes the aforementioned limitation since the features extractor is trained on the same dataset as the generator and thus extract relevant features. 
 


	To show that, we trained our different networks (i.e., with different features extractors) on three distinct datasets (examples of images of these datasets are shown in Figure <ref>):
    
    
  * A subset of ImageNet <cit.>, for which we sampled 70,000 images. Since VGG19 was trained on ImageNet for many (more than 300K) iterations, we expect to have similar or worse results than the state-of-the-art method SRGAN from <cit.> on this database.
    
  * The Describable Textures Dataset (DTD) <cit.>, containing 5,600 images of textural patterns. These data are relatively close to ImageNet and we show that our method gives convincing results relatively close to SRGAN.
    
  * A dataset containing satellite images[Can be found in <http://www.terracolor.net/sample_imagery.html>], which we generated by randomly cropping 256× 256 images on a 7205× 7205 satellite image which result in 235,183 images. We particularly show that our method significantly outperforms SRGAN on this dataset. We refer to this dataset by Sat.
    
 
All experiments are performed with a scale factor of 4× between low- and high-resolutions images and the formers are obtained during the training by down-scaling the original images by a factor 1/4.



  §.§.§ Architectures

	Our overall goal is to prove that the proposed GCN framework, is adapted to train a generative mapping model and that it surpasses the MSE loss in keeping perceptual similarity in the generated image (whereas the MSE loss tends to smooth things out and lose high frequency details). As opposed to <cit.>'s work, our framework does not require to have a pre-trained network, like VGG, to extract helpful features for training. In this paper, we focus on the Super Resolution problem. Therefore, we chose our mapping function ℱ, or generator, to be that of Ledig et al. <cit.>: a feed-forward CNN parametrized by θ_ℱ, composed of 10 residual blocks. These blocks are made of two convolutional layers with 3× 3 kernels and 64 features maps, each followed by batch normalization and PReLU as activation. The image's size is then increased of a factor 4 by two trained upsamplings. The architecture of all the used discriminators follows the guidelines of Radford et al. <cit.> as it is composed of convolutional layers, followed by a batch normalization and a LeakyReLU (α = 0.2) activation. This block is repeated eight times and each time the number of 3× 3 kernels increases by a factor 2 (ranging from 64 to 512), a strided convolution is used to reduce the image resolution by 2. Two dense layers and a sigmoid activation then return the discrimination probability. In the case of an auto-encoder (every Reconstruction problem), we follow the same architecture for the encoder and a symmetric one for the decoder. Figure <ref> depicts an overview of the architectures for both the generator and the discriminator.
	
	
  


  §.§.§ Training details and parameters

	All networks were trained[A Keras implementation is provided in <https://github.com/melaseddik/GCN>] on a NVIDIA Geoforce GTX 1070 GPU using the datasets described in Section <ref>, which do not contain the (1000) testing images shown as results.  We scaled the range of both the LR input images and the HR images to [-1, 1], which explains the tanh activation for the last layer of the generator. All variants of our networks, which differ in their features extractor, were trained from scratch (for the generator and the features extractor) with mini batches of 10 images. We used the Adam optimizer with a learning rate of 2· 10^-4 and a decay of 0. The generator and the feature extractor are updated alternatively. As we realized training was stable and quite fast, we trained with only 5,000 update iterations to pinpoint the best method among the different GCNs. Finally, the regularization parameters in our global loss are set by default as λ_1=1 and λ=10^-3. As a reminder, our goal here is, given a generator architecture (or mapping function ℱ), to find the best strategy to train it, following our GCNs paradigms. The best method is then further compared to baselines.



  §.§.§ Features Extractor Selection


	As we said above, we investigated the ability of different features extractor to construct relevant perceptual feature maps for training and improving the rendering quality of the generator. In order to select the best learning strategy given a certain dataset, we train the generator on each dataset (presented in Section <ref>) using the different learning strategies: 𝒫_sisr/rec, 𝒫_sisr/dis, 𝒫_sisr/dis,rec, 𝒫_sisr/adv and 𝒫_sisr/adv,rec. Note that, the features extractor for all the considered methods correspond to the first layer of the discriminators (or encoder-decoders). In fact, as the problem 𝒫_sisr consists in recovering low-level perceptual cues, we limited our study to the first layer.  

	Table <ref> summarizes the results of the proposed 𝒫_sisr methods in terms of low-level metrics (L2 and SSIM) and perceptual metrics <cit.> which are given by Eq. (<ref>). We notice from this table that the method 𝒫_sisr/adv,rec performs relatively well on the datasets ImageNet and Sat in terms of perceptual metrics. While 𝒫_sisr/dis,rec gives better results on the DTD dataset. The main difference between these two methods is that the former considers an adversarial loss on the objective function while the latter does not consider the adversarial term. This explains the reason why 𝒫_sisr/adv,rec does not perform well on DTD. In fact, texture images belong to a complex manifold and their distribution is relatively hard to fit by a generative model. 
    
    Figure <ref> shows qualitative results of the different proposed methods on the different presented datasets. Generally, the methods which were trained with an additional adversarial loss (𝒫_sisr/adv and 𝒫_sisr/adv,rec) output images of higher quality (on the datasets ImageNet and Sat) as GANs were introduced to do just so: generate images that follow the distribution of the dataset. Among these two adversarial methods, it seems to us (as suggested by the quantitative results of table <ref>) that 𝒫_sisr/adv,rec (column (c) of Figure <ref>) is able to detect and render more details, due to its ability to generate more relevant features as the features extractor Φ is learned to solve a multi-task problem; namely a discrimination and a reconstruction problem, in particular, this method allows for the learning of both classification and reconstruction-based features. We will thus further investigate the 𝒫_sisr/adv,rec method for the comparison to the baseline and the state-of-the-art method SRGAN <cit.>, on the satellite images domain.



