Structure-Preserving Image Super-resolution via Contextualized Multi-task Learning
Yukai Shi Keze Wang Chongyu Chen Li Xu Liang Lin
cs.CV




𝐱
ŁL






IEEE Transactions on Multimedia
Shi et al.:


Structure-Preserving Image Super-resolution via Contextualized Multi-task Learning
    Yukai Shi,
    Keze Wang,
    Chongyu Chen,
    Li Xu, and
    Liang Lin

Yukai Shi, Keze Wang, Chongyu Chen and Liang Lin are with School of Data and Computer Science, Sun Yat-sen University, Guangzhou, China. Keze Wang is also with Dept. of Computing, The Hong Kong Polytechnic
University, Hong Kong. Li Xu is with SenseTime Group Limited, China. Contact: chenchy47@mail.sysu.edu.cn. Liang Lin is also with the Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, Sun Yat-sen University, China.


    
=======================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================


Single image super resolution (SR), which refers to reconstruct a higher-resolution (HR) image from the observed low-resolution (LR) image, has received substantial attention due to its tremendous application potentials. Despite the breakthroughs of recently proposed SR methods using convolutional neural networks (CNNs), their generated results usually lack of preserving structural (high-frequency) details. In this paper, regarding global boundary context and residual context as complimentary information for enhancing structural details in image restoration, we develop a contextualized multi-task learning framework to address the SR problem. Specifically, our method first extracts convolutional features from the input LR image and applies one deconvolutional module to interpolate the LR feature maps in a content-adaptive way. Then, the resulting feature maps are fed into two branched sub-networks. During the neural network training, one sub-network outputs salient image boundaries and the HR image, and the other sub-network outputs the local residual map, i.e., the residual difference between the generated HR image and ground-truth image. On several standard benchmarks (i.e., Set5, Set14 and BSD200), our extensive evaluations demonstrate the effectiveness of our SR method on achieving both higher restoration quality and computational efficiency compared with several state-of-the-art SR approaches. [The source code and some SR results can be found at: <http://hcp.sysu.edu.cn/structure-preserving-image-super-resolution/>]



Structure-preserving Image super-resolution; Convolutional network;  Context learning; Multi-task learning;





§ INTRODUCTION


Image super-resolution (SR) is a fundamental problem in image processing.


Single image SR approaches, which aim at restoring a high-resolution (HR) image only from a single low-resolution (LR) image, have been applied to many 
image and video analysis tasks, such as
video surveillance <cit.>,
image-based medical analysis <cit.>, 
and image/video streaming <cit.>.

Common techniques for single image SR can be roughly categorized into reconstruction-, example- and interpolation- based approaches. Reconstruction-based approaches <cit.>, which restore HR images by deconvolutional methods <cit.> with a global blur degradation model, usually introduce ringing artifacts around salient structures <cit.> due to inaccurate blurring kernels in the inverse problem. Example-based approaches <cit.> boost the amplification factor by using internal or external patch data to guide the image restoration. Recently, Huang et al. <cit.>
 proposed to exploit self-similarity for single image SR,
which greatly expands the internal patch searching space. Hu et al. <cit.> proposed a cascaded linear regression technique to model the relationship between HR and LR images. Interpolation-based approaches can achieve acceptable trade-off between performance and efficiency with a pre-defined kernel. However, pre-defined kernels use fixed weights for interpolation, which will inevitably cause blur when the weight definition is inconsistent with image structures. To address issue, various adaptive interpolations <cit.> are proposed. But the improvements in restoration quality are still limited.

The success of deep convolutional neural network (CNN) in computer vision tasks has inspired novel trends in low-level image restoration researches, such as rain/dirt removal <cit.>, noise removal <cit.>, face hallucination <cit.>, hashing <cit.> and image inpainting <cit.>. Focusing on learning an end-to-end mapping between the LR images and their corresponding HR images, several CNN-based methods <cit.> have been proposed to perform image SR in a pure data-driven manner. That is, they directly minimize the mean squared error (MSE) between the predicted and ground-truth images in the training stage.
Although the restoration performance is significantly improved, the structural
inconsistency between the LR input and HR output still exists. 
This is because human visual system is more sensitive to structural changes,
which are difficult to be exploited from MSE-based loss functions.
Recent advances in image SR try to address this
issue <cit.> by introducing
feature-based perceptive loss functions to the training stage. However, unwanted artifacts and unreal details are also introduced, which make their SR results look unrealistic.






Considering single image SR is an ill-defined problem, it is necessary to exploit the priors
of natural image to further improve the SR performance.
Motivated by recent advances in deep learning researches


that exploit priors in the form of context information in designing
neural networks <cit.>,
in this work, we propose to design neutral networks to investigate two types of image
structural information, i.e., global structural information which corresponds to salient
boundaries in a global perspective and residual structural information which contains
noticeable details that are critical to visual quality. 
The success of multi-task learning framework inspires us
to leverage such structural information in a unified manner. For instance, Yang et al. <cit.> proposed to utilize the common knowledge (e.g., feature selection functions) of multiple tasks as supplementary information to facilitate decision making. Considering aforementioned structural information are usually considered as complementary context rather than common knowledge, in this work, we concentrate on complimentary contextualized multi-task learning for structure-preserving single image SR.
In particular, we propose a deep
joint contextualized multi-task learning framework, where three types of image components are imposed as complimentary contexts and jointly learned,
i.e., the base image content, the boundary map, and the residual map.

Besides a convolutional network that learns content-adaptive interpolations to produce
the intermediate base image, we impose an auxiliary task to back-propagate the
global boundary structural context.



Meanwhile, an independent sub-network is introduced to explicitly model
the noticeable details to provide residual structural context.



















The major contribution of this work is the proposed contextualized multi-task learning
framework, which is the first attempt to incorporate joint learning
of local, global, and residual contexts into CNNs for single image SR.
Other contributions mainly come from the proposed content-adaptive interpolation
and the sub-networks for capturing complementary image contents, which enables
better trade-off between restoration quality and the number of network parameters. 

Extensive experiments on several benchmarks datasets (e.g. Set5, Set14, BSD500) demonstrate that the proposed framework shows superior performance to most learning-based approaches in the perspective of both visual quality and quantitative metrics, which
facilitates the real-time image SR process.



We would like to point out that a preliminary version of this work is reported
in <cit.>, which coarsely concatenates content-adaptive interpolation
and holistic edge context. 

In this paper, we inherit the idea of preserving structures and refining the network architecture.
A simple yet powerful sub-network is further employed to capture noticeable
image details for better visual quality.
The whole framework is re-interpreted from the aspect of joint context learning and
multi-task learning.

Besides, more comparisons with state-of-the-art approaches and more detailed analyses
of the proposed modules are added to further verify our statements.




The rest parts of this paper are organized as follows. Section <ref> briefly reviews existing machine learning-based SR approaches which motivate this work. Section <ref> presents the details of the proposed framework, with thorough analysis of every component. Section <ref> demonstrates the experimental results on several public benchmarks, comparing with state-of-the-art alternatives. Finally, Section <ref> concludes this paper.




§ RELATED WORK






 §.§ Interpolation-based image super-resolution


Interpolation-based approaches typically start from evenly placing the pixels of LR image to the HR grid (the integral coordinates in the HR image domain).
The basic idea of these approaches is to estimate the unknown pixel values in
the HR grid by weighted average of surrounding known pixels.
Considering common pixel changes in a local region can be approximated by continuous functions, 
people have proposed various weight definitions for image interpolation. For example, bilinear 
interpolation is proposed to utilize local linearity, and bicubic interpolation is proposed to exploit the
high-order continuity <cit.>. 
However, there are plenty of pixel changes that cannot be described by these pre-defined functions,
especially for regions with rich image structures. In this case, structures will be blurred
due to improper pixel averaging. To address this problem, various adaptive interpolation <cit.> are proposed. For instance, Walt et al. <cit.> proposed to express polygonal pixel overlap as a linear operator to improve the interpolation performance. But the improvements are still limited.







 §.§ Multi-task learning in image super-resolution

Decades of researches on multi-task learning have demonstrated that learning multiple correlated tasks simultaneously can significantly improve the performance of the main task <cit.>. 
In single image SR, there is also a trend of utilizing multi-task learning.
For example, Yang et al. <cit.> proposed a multi-task K-SVD 
learning for image SR, in which example image patches are divided into different groups
and K-SVD is applied to every group.
It is shown that simultaneous learning multiple dictionaries can lead to better SR quality.
Liang et al. <cit.> proposed a multi-task learning framework that jointly
considers image SR process and the image degeneration process.


These works claim that the multi-task learning framework is a feasible way of utilizing priors in learning-based image SR. 



 §.§ Deep learning in image super-resolution

Recently, deep learning has achieved




significant quality improvements in image SR.
For example, Dong et al. <cit.> utilized a three-layer fully convolutional
network to learn the non-linear mapping between HR and LR patches,
which has a close relationship to sparse coding. 
Ren et al. <cit.> introduced Shepard CNNs
to facilitate translation variant interpolation, which gives a solution to both inpainting and SR.


Wang et al. <cit.> proposed a sparse coding based network for
image SR. Based on learned iterative shrinkage and thresholding algorithm(LISTA) <cit.>, they employ a set of neural networks to restore images.
Zeng et al. <cit.> proposed a deep autoencoder for SR,
which explores the consistent representations of HR and LR images and 
demonstrate a superior efficiency compared to similar methods based on sparse
representation. 
Kumar et al. <cit.> studied on several factors that affect the
training phase 

to facilitate learning-based SR with fewer training samples.
The models of these methods, although being proposed from different aspects, are trained 
to minimize the squared error w.r.t. the ground-truth HR image, which is not necessarily correlated
to good perceptual quality. Bruna et al. <cit.> referred this problem
as regression to mean. Their proposed solution is a conditional generative model,
which demonstrates improvement over visual quality, but with high time cost in both training
and testing.











More recently, researchers notice the importance of image details and make various of attempts for exploration. Kim et al. <cit.> further improved the SR quality by different network architectures such as very deep and recursive network structures. However, these methods heavily rely on very deep networks with plenty of parameters. e.g., a 20-layer convolutional neural network <cit.>. In addition, perceptual losses have
been proposed for CNNs <cit.>, which conduct the loss from the image space to high-level feature space of a pre-trained VGG-net <cit.>. 
At the same time, Ledig et al. <cit.> proposed to apply adversarial network to the task of SR, which results in more image details but lower PSNR score.
More related to our work, there are several attempts to accelerate image SR. By developing a sub-pixel convolutional layer, Shi et al. <cit.> used a single model to handle real-time image SR. Similarity, Dong et al. <cit.> applied convolutional layers on LR image and upscaled it with deconvolution. They both promise low computational complexity, but there still exists plenty of room for performance improvement.




§ CONTEXTUALIZED MULTI-TASK LEARNING
 






In this section, we present the details of our framework. As sketched in Fig. <ref>, the proposed framework includes three components: feature extraction, content-adaptive interpolation, and multi-task estimation. 












 §.§ Feature Extraction

Inspired by the Pyramid-Net <cit.>, we design a pyramid network
structure for feature extraction.
That is, there are 3 convolutional layers with 

16, 32 and 128 kernels, respectively.
Detailed setup is summarized in Table <ref>.


The first layer with kernel size 5×5 is designed as a spacious receptive field to capture as much image information as possible, as illustrated in <cit.>.

The other two layers with 3×3 kernel are adopted for better efficiency
as <cit.>. Note that we focus on extracting features 
from original LR images instead of the interpolated images.
Thanks to the decreased computations of convolutional operations caused by the small size of feature maps,
the proposed feature extraction can significantly accelerate the speed without
obvious quality drop.
Since the LR image has been represented as high-dimension feature maps through
the first 3 layers, the computation cost may become pretty high if we
import the high-dimension feature maps to content-adaptive interpolation directly.

Therefore, we apply a shrinking layer with 8 kernels of size 1×1 to reduce
the feature dimension. Note that the kernel number is empirically chosen for a reasonable trade-off between effectiveness and efficiency.


Benefitting from the shrinking layer, our model not only avoids parameter explosion but also promotes the restoration efficiency.






 §.§ Content-adaptive Interpolation

The second component is one deconvolutional layer, which is used to interpolate the LR feature maps in a content-adaptive way. The deconvolutional layer has 8 kernels of size n × n.

Note that in this work, n is determined by the upscaling factor,
which follows the principles of bicubic interpolation.
That is, the kernel should be large enough to cover the second pixel
around the anchor pixel in the HR grid.
For example, the deconvolutional kernel is of size 8×8,
11×11, and 16×16 for the upscaling factors of
2, 3 and 4, respectively. 
In this way, the deconvolutional layer can be regarded as
a neural network implementation of
standard image interpolation. Let 𝐲 be the HR image with a HR grid.
We construct another HR image 𝐱 by evenly placed the LR image in 
the HR grid with identical pixel intervals. Then, standard interpolation can be written as:

    𝐲_j = ∑_i ∈Ω_j𝐱_i ω_ji,

where i and j are the pixel indices in the HR grid, 
Ω_j represents the subset of n× n neighbouring pixels around pixel j,
and ω_ji is the pre-defined weight for interpolation. 
Note that 𝐱_i is non-zero only when it comes from a pixel in the LR image.

With these definitions, we re-formulate the interpolation process
as a basic component of a deconvolutional layer, i.e.,

    𝐲_j = δ (∑_i∈Ω_j𝐱_i W(i') + b)

where δ(·) represents the activation function, W is the deconvolutional kernel,
i' represents the pixel of W that contributes to pixel j, and b is the bias.

















In the proposed content-adaptive interpolation, we use multiple deconvolutional kernels in a similar fashion. 


That is, we evenly place the LR image in the HR grid to construct 𝐡^l. Then,

    𝐡^l+1_k = δ ( 𝐡^l⊗ W_k
    + b_k ),

where the subscript k represents the kernel index, “⊗” represents the
convolutional operator, and 𝐡^l+1 is the output image of the
l^th layer.

In this way, content-adaptive image interpolation can be accomplished via a deconvolutional layer,
whose kernels are learned from sufficient training data.
Note that the deconvolutional layer is in the middle of the proposed network, which is different from other CNN-based SR methods <cit.> that use deconvolution as the last layer. It is shown empirically that the proposed network can achieve nice restoration quality with reasonably increasing network parameters.

To compare the proposed network with the bicubic interpolation, we construct a small network which only has one deconvolutional layer to learn an adaptive kernel, taking BSD300 as training data and bicubic interpolation parameters for initialization. The intensity changes of bicubic and our learned kernels are visualized in Fig. <ref>, which illustrates that the learned kernel contains more high-frequency components. Meanwhile, the restoration results also indicate that the learned kernel leads to a superior restoration quality with more recovered details compared to the bicubic kernel. Thus, the effectiveness of the proposed adaptive interpolation is verified.



 §.§ Contextualized Multi-task Learning




















In spired by the multi-task learning principles, we make an attempt to introduce auxiliary knowledge to SR issue.



Global Boundary Context: We develop a Boundary Context sub-Network (BCN) to preserve salient boundaries that represent global image structures.
BCN consists of two convolutional layers with 3×3 kernels, where one layer is with 12 kernels and the other layer is with 2 kernels. In the training phase of BCN, we propose to exploit salient image boundaries by regarding edge detection as a joint task of HR image restoration. In particular, we introduce an auxiliary term into the objective function, which computes the error between predicted and human-labeled edge/boundary maps. These boundary maps are from Berkeley Segmentation Dataset (BSD) <cit.>. Note that there are multiple boundary maps in BSD500 data set, we use their summation for better visualization and show the examples in Fig.<ref>. 

With the two tasks of image restoration and edge detection, image components and
structural features are firstly extracted and enlarged by content-adaptive interpolation before being fed into the BCN. Several representative samples of the extracted feature maps are shown in Fig. <ref>, in which the top row and bottom row show image-like and edge-like features, respectively. This implies that these layers simultaneously extract redundant components and features, making it possible to produce base image and boundary maps in the HR image domain. 

Through joint optimization in an end-to-end manner, feature extraction, content-adaptive interpolation and BCN can provide complimentary context information to each other. In this way, structure-aware feature representations can be learned with the content-adaptive interpolation. 

Residual Context: As a result of paying close attention to generating the HR image with salient boundaries, the concatenated BCN might fail to restore some subtle but noticeable structures. Motivated by the recent residual learning paradigm <cit.>, we make an attempt to address this issue by employing a Residue Context sub-Network (RCN). The objective of the RCN is to synthesize a residual image, which is defined as the difference between the interpolated HR image and the ground-truth HR image. In contrast to using the bicubic interpolated HR image as in <cit.> and <cit.>, our model uses the intermediate HR image provided by BCN. This can bring us two benefits: i) Higher image SR performance. As the HR image provided by BCN achieves comparable performance to the state-of-the-art methods, RCN can focus on remedying the overlooked information for higher SR quality; ii) A lightweight network architecture for RCN. Our used interpolated image contains significantly richer information than the bicubic one. Hence, compared with <cit.> and <cit.>, the synthesization of residual images is much easier. As illustrated in Fig. <ref>, the architecture of RCN is the same as that of the concatenated BCN.


For the joint optimization of content-adaptive interpolation, BCN and RCN, we develop a fusion layer to merge the intermediate output of RCN and BCN in a data-driven way. In particular, the final HR image 𝐲 of our framework is obtained by:

    𝐲=𝐟⊗𝐈_interHR + 𝐈_r,

where 𝐟 denotes a 3×3 convolutional filter, 𝐈_interHR is the intermediate HR image provided by BCN, and 𝐈_r is the residue image synthesized by RCN. In this way, the parameters of 𝐟 can be adaptively updated during the learning process. 




§ FRAMEWORK TRAINING

The proposed framework is jointly optimized on a set of “LR image, HR image and HR edge
map[In BSD data sets, more than one boundary maps are provided for every image, which
are all used in our training process. Since multiple boundary maps are used in the same way, in this
subsection, we focus on the case of one boundary map for simplicity.]” triplets.
For convenience, we use 𝐈_l,  𝐈_h and 𝐈_b to represent
the LR image, HR image and boundary map, respectively.
Given the input 𝐈_l, the objective of our model is to reconstruct a HR image similar to 𝐈_h and predict a boundary map similar to 𝐈_b.

The parameter 𝐖 of our model can be divided into 4 disjoint parts, i.e., 𝐖={𝐖_s, 𝐖_h, 𝐖_b, 𝐖_d}, where 𝐖_s and 𝐖_d denote the parameters of content-adaptive interpolation and RCN, respectively. We denote the parameter of feature extraction stage has combined into content-adaptive interpolation part. For BCN, we use 𝐖_h and 𝐖_b to represent the specific weights for generating the intermediate HR image and the boundary maps, respectively.
Since the parameters are separable, we propose to train our model in three iterative steps.
First, we jointly train content-adaptive interpolation and BCN until their convergence; Second, fixing the parameters of content-adaptive interpolation and BCN, we update the parameters of RCN. Third, we jointly optimize content-adaptive interpolation, BCN and RCN. Specifically, content-adaptive interpolation and BCN are trained according to the following objective function:

    [       L(𝐈_l,𝐈_h, 𝐈_b, 𝐖)  =   L_h(𝐈_l,𝐈_h, 𝐖_s, 𝐖_h)  +;                             α· L_b (𝐈_l,𝐈_b, 𝐖_s, 𝐖_b), ]

where L_h and L_b represent the HR image reconstruction objective and the boundary prediction objective, respectively.
The balance weight α is used to control the importance of L_h and L_b, which is empirically set to 1 in all our experiments.
Both L_h and the L_b are in the form of mean squared error (MSE), i.e.,

    L_h = 1/N∑_i=1^N ( 𝐈_h^i - f_h( 𝐖_s, 𝐖_h, 𝐈_l^i)  ) ^ 2,

and 

    L_b = 1/N∑_i=1^N ( 𝐈_b^i - f_b( 𝐖_s, 𝐖_b, 𝐈_l^i)  ) ^ 2,


where f_h(·) and 
f_b(·) 
denote the reconstructed HR image and the predicted boundary map, respectively,
i represents the sample index, and N is the number of training triplets.
For simplicity, we use 𝐈_ω to denote f_b( 𝐖_s, 𝐖_b, 𝐈_l).

Note that when multiple boundary maps are available, there will be more edge prediction objectives.











The loss function for training RCN is defined as:

    L_d =1/N∑_i=1^N(𝐈_h^i-𝐈_ω^i-f_d(𝐖_s,𝐖_d, 𝐈_l^i))^2.




Finally, the whole framework is optimized by employing the standard back propagation algorithm, i.e., 

    L = 1/N∑_i=1^N (𝐈_h^i - y)^2,

where y, the output of fusion layer, is the final HR image in the testing phase.

The whole training phase is summarized as Algorithm <ref>,

which accords with the pipeline of our proposed framework in Fig. <ref>.




§ EXPERIMENTS





 §.§ Experiment Setting

Datasets: All experiments are evaluated on three challenging benchmarks, i.e., Set5 <cit.>,
Set14 <cit.> and BSD500 <cit.>. The BSD500 dataset consists of 500 natural images and human annotations for corresponding boundaries. We use the 300 images from its training and validation set for training.
The rest of 200 images in BSD500 dataset form a widely used benchmark called BSD200. Besides, the Set5 and Set14 datasets are also adopted
as testing sets in other state-of-the-art methods such as <cit.>. Thus, we conduct experiments on the three
benchmarks.






Implementation details: In the training phase, we first convert the original color image to grayscale image by extracting the luminance component in YCbCr color space.
Then, we downscale the training images by requested scaling factors (e.g., 2, 3, and 4) to obtain the LR images. The LR images are cropped into a set of patches with a stride of 4.  The size of patches is set to be same as receptive field.
The corresponding HR images and boundary maps are cropped with respect to the scaling factors.
Before training, we initialize the network parameters by a zero-mean Gaussian distribution with a standard deviation of 1×10^-4. 
For the pre-training of the proposed model, we use the 91-images <cit.>
and PASCAL VOC2012 <cit.> datasets, which totally contain 13,487 clear images.
Specifically, the model using LR and HR image pairs is pre-trained following the same strategy as <cit.>. Since the feature extraction stage employ pyramid structure, we speed it up with the help of Factorized CNN <cit.>. In the training on BSD300 dataset, The learning rate of the last layer is set to 1×10^-5, while the rest layers are using a fixed learning rate of 1×10^-4.

To increase the number of training samples, we also employ data augmentation for BSD300
dataset, as reported in <cit.>.





Methods and metrics: We compare our model with several recent state-of-the-art methods, including a three-layer CNN (SRCNN) <cit.>, super-resolution forest (SRF) <cit.>, sparse coding-based network (SCN) <cit.>, anchored neighborhood regression (A+) <cit.>, shepard interpolation neural network (ShCNN) <cit.>, very deep convolutional network (VDSR) <cit.>, and fast convolutional network for SR (FSRCNN) <cit.>.
For fair comparisons, we employ the popular PSNR and SSIM metrics for evaluation.
To evaluate the structure-preserving capability, we introduce a new metric called “EPSNR”, which can be formulated as:

    EPSNR = 10log_10(MAX_I^2/1/|E|∑_i ∈ E(G_i-P_i)^2),

where MAX_I=255 is used for 8-bit images, G and P denote the ground-truth and the produced HR images, respectively, E indicates the pixels whose distances to their closest boundary are less than 2 pixels, and i is the pixel index. It is believed that EPSNR can better exploits image fidelity on edge regions.









We have also investigated the model complexity from the aspect of parameter number. Two profiles of our model are used, i.e., the common model (denoted as “ours”) used in the above comparisons, and the model with a much deeper architecture (denoted as “deeper ours”). In the “deeper ours” profile, we only increase the convolutional layer number of feature extraction stage from 4 to 18. Thus our model has similar number of parameters compared to VDSR. Both profiles can be accelerated by cuDNN <cit.>. All the CNN-based methods are compared using the Set5 dataset with a scaling factor of 3. The results illustrated in Table <ref> demonstrate that the performance of our model keeps increasing as the parameter number increases. Using comparable network parameters, our model can achieve a PSNR gain of 0.14 dB compared to VDSR. 
Since fewer parameters can benefit both the training and testing phases, we recommend our model with the common profile. Fig. <ref> illustrates the efficiency of all the compared methods using the “time-quality” diagram. It is demonstrated that our model with common profile runs nearly 2 times faster than VDSR while maintaining the second best SR performance, which is quite suitable for lightweight and fast implementation on consumer-grade devices.
For applications that require extremely high SR quality, deeper ours will be a nice choice.



Some promising examples are visualized in Fig. <ref> and Fig. <ref>. For better viewing, we interpolate the chrominance components by the bicubic method to generate color images. To clearly demonstrate the difference, we choose one 
patch from each image and attach them below. Compared to other methods, our model can produce images with sharper and clearer boundaries. 

Visual Comparison with SRGAN: We compare our method with the super-resolution generative adversarial network (SRGAN) <cit.>. Because of their proposed adversarial loss, SRGAN has obtained promising performance. However, it still has 
 problems in recovering real details, which is verified by
the comparisons shown in Fig. <ref>. It is shown in the enlarged patches of Fig. <ref> (c) and (d) that some waterdrops exist in the ground-truth image disappear, which are produced by SRGAN methods. But these waterdrops are captured by our method and ShCNN. 
As pointed out in <cit.>, SRGAN tends to bring in similar textures instead of recovering real details.

Therefore, our proposed framework performs better than SRGAN on recovering more accurate details.


Discussion on real-world cases: To justify the effectiveness of our method, we move one step forward to deal with images from video surveillance and mobile device. Specifically, we apply our model on real-world images with a scaling factor of 3. As reported in Fig. <ref>, “Original” indicates the original images and “Proposed” represent the images processed with our model. As one can observe from results shown in Fig. <ref>, “Proposed” have fewer artifacts compared with “Original”. This demonstrates the robustness of our method towards real-world challenges. 



 §.§ Ablation Study
 
In this subsection, we conduct detailed analyses on the proposed modules, i.e., content-adaptive interpolation, BCN and RCN, for better understanding of our framework. 
We hope such analysis can lead to new insights into image restoration researches.

Content-adaptive interpolation:
One of the major differences between our model and SRCNN <cit.> is the employment of the deconvolutional layer. To demonstrate the superiority of our design,
we train several fully convolutional networks (FCNs) with various layer numbers for comparisons. Specifically, we increase the number of middle layers from 5 to 16, resulting in FCN-5, FCN-9, FCN-12, and FCN-16. These FCNs follow the bicubic upsampling strategy as in SRCNN <cit.>. Our content-adaptive interpolation consist of 5 convolutional layers and one deconvolutional layer, which contain feature extraction stage, content-adaptive interpolation and BCN. We remove the task of boundary objective to address the effectiveness of content-adaptive interpolation. By comparing content-adaptive interpolation with these FCNs on Set5 dataset with a scaling factor of 3, we obtain the results shown in Table <ref>.



It is indicated in these results that although the SR performance of FCN keeps increasing as the network depth increases, it still cannot outperform content-adaptive interpolation even when there are 16 layers.


Nevertheless, our content-adaptive interpolation network, which only has 6 layers, surpasses these FCNs by a clear margin. More specifically, content-adaptive interpolation network outperforms FCN-16 by 0.32 dB. This explicitly verifies the superiority of the content-adaptive interpolation.



Global Boundary Context:
The proposed BCN is motivated by the paradigm of mult-task learning, which incorporates edge estimation as a co-task of HR image generation. Therefore, its analysis is conducted by comparing the SR performance between with and without the edge prediction objective.

Since the BSD200 dataset contains manually labeled boundary maps, based on which we can easily compute the EPSNR. We compare two profiles of our model on this dataset with a scaling factor of 3 using both PSNR and EPSNR metrics. 
By removing the boundary prediction objective, we degrade BCN into single-task learning and denote it as “ours w/o boundary”. As illustrated in Table <ref>, the PSNR and EPSNR gains indicate the benefit of multi-task learning. 
Because the boundaries only occupy a small portion of the whole image, the improvement on overall PSNR is minor. However, the large improvement on EPSNR verifies the effectiveness of BCN.
Another benefit of incorporating boundary prediction objective is the acceleration of training process. 

As shown in the PSNR curves of Fig. <ref>, the edge prediction objective not only accelerates the convergence, but also contributes to a higher restoration quality. 






Local Residue Context:
We design RCN to provide complementary information for image SR. Therefore, the SR performance
of our model will be degraded if RCN is removed.
To verify our statement, we use another profile named “ours w/o RCN”, which is very similar to the previous version of this work <cit.>,  to conduct more comparisons on the aforementioned datasets with a scaling factor of 3. 

Table <ref> reports the comparison results. It is shown that, although content-adaptive interpolation and BCN can produce HR image of high quality, the SR performance can still be further improved.

The improvement on PSNR is minor because PSNR is a squared error-based metric, which is difficult to reveal subtle structure differences. In contrast, because SSIM concentrates on structure similarity, the improvement on SSIM is more significant.





§ CONCLUSION AND FUTURE WORK
 
In this paper, to address single image super-resolution, we have proposed a novel contextualized multi-task deep learning framework. Our neural network model incorporates global boundary context and residual context to super-resolve images while well preserving their structural details. 
 Moreover, we have introduced “content-adaptive interpolation", which leverages a set of filters that are adaptive to the training samples. Different from the kernel estimation in blind image SR which usually employs only a single filter, our proposed content-adaptive interpolation has more filtering parameters and better convenience of being embedded into CNNs. Our extensive experiments suggest that the proposed method outperforms other leading image super-resolution approaches, and achieves state-of-the-art performances on both popular evaluation metrics and visual quality comparison. 

There are several directions to extend our method. First, we are considering to introduce a perceptual loss into the multi-task optimization, aiming to better capture realistic and meaningful image details. Second, we shall generalize this framework to adapt to video data by taking spatio-temporal coherency into consideration.
Third, considering that additional common knowledge in deep neural networks would be an interesting trial, we intend to utilize complementary spatial-temporal contexts as privileged information for video SR, as suggested by Yang et al. <cit.>.




§ ACKNOWLEDGEMENTS

This work is partially supported by NSFC (No. 61602533), The Fundamental Research Funds for the Central Universities, in part by Hong Kong Scholars Program and Hong Kong Polytechnic University Mainland University Joint Supervision Scheme. We are grateful to acknowledge NVIDIA for GPU donations.






















IEEEbib


[
    < g r a p h i c s >
]Yukai Shi received his B.E. degree from Heilongjiang University. He is currently working toward the Ph.D. degree with the School of Data and Computer Science, Sun Yat-Sen University. His research interests including computer vision and machine learning.


[
    < g r a p h i c s >
]Keze Wang received his B.S. degree in software engineering from Sun Yat-Sen University, Guangzhou, China, in 2012. He is currently pursuing the dual Ph.D. degree at Sun Yat-Sen University and Hong Kong Polytechnic University, advised by Prof. Liang Lin and Lei Zhang . His current research interests include computer vision and machine learning. More information can be found in his personal website http://kezewang.com



[
    < g r a p h i c s >
]Chongyu Chen is a distinguished associate research fellow of Sun Yat-sen University. He received his B.S. and Ph.D. degrees from Xidian University, Xi'an, China, in 2008 and 2014, respectively. From 2015 to 2017, he was a post-doctoral fellow at the Hong Kong Polytechnic University. His research interests include image restoration, 3D computer vision, signal separation, etc.



[
    < g r a p h i c s >
]Li Xu received the BS and MS degrees in computer science and engineering from Shanghai JiaoTong University (SJTU) in 2004 and 2007, respectively, and the PhD degree in computer science and engineering from the Chinese University of Hong Kong (CUHK) in 2010. He joined Lenovo R & T Hong Kong in Aug 2013, where he leads the imaging & sensing group in the Image & Visual Computing (IVC) Lab. Li received the Microsoft Research Asia Fellowship Award in 2008 and the best paper award of NPAR 2012. His major research areas include motion estimation, motion deblurring, image/video analysis and enhancement. He is a member of the IEEE. 



[
    < g r a p h i c s >
]Liang Lin is a full Professor of Sun Yat-sen University. He received his B.S. and Ph.D. degrees from the Beijing Institute of Technology (BIT), Beijing, China, in 2003 and 2008, respectively, and was a joint Ph.D. student with the Department of Statistics, University of California, Los Angeles (UCLA). From 2008 to 2010, he was a Post-Doctoral Fellow at UCLA. From 2014 to 2015, as a senior visiting scholar he was with The Hong Kong Polytechnic University and The Chinese University of Hong Kong. His research interests include Computer Vision, Data Analysis and Mining, and Intelligent Robotic Systems, etc. He has authorized and co-authorized on more than 100 papers in top-tier academic journals and conferences. He has been serving as an associate editor of IEEE Trans. Human-Machine Systems. He was the recipient of the Best Paper Runners-Up Award in ACM NPAR 2010, Google Faculty Award in 2012, Best Student Paper Award in IEEE ICME 2014, Hong Kong Scholars Award in 2014 and The World's First 10K Best Paper Diamond Award in IEEE ICME 2017. More information can be found in his group website http://hcp.sysu.edu.cn.


