Gated Fusion Network for Joint Image Deblurring and Super-Resolution
Xinyi Zhang Hang Dong Zhe Hu Wei-Sheng Lai Fei Wang Ming-Hsuan Yang


Julian Schwinger — Recollections from many decades
    Stanley Deser
    
==================================================





Single-image super-resolution is a fundamental task for vision applications to enhance the image quality with respect to spatial resolution. 

If the input image contains degraded pixels, the artifacts caused by the degradation could be amplified by super-resolution methods.

Image blur is a common degradation source.

Images captured by moving or still cameras are inevitably affected by motion blur due to relative movements between sensors and objects. 

In this work, we focus on the super-resolution task with the presence of motion blur.

We propose a deep gated fusion convolution neural network to generate a clear high-resolution frame from a single natural image with severe blur. 

By decomposing the feature extraction step into two task-independent streams, the dual-branch design can facilitate the training process by avoiding learning the mixed degradation all-in-one and thus enhance the final high-resolution prediction results.

Extensive experiments demonstrate that our method generates sharper super-resolved images from low-resolution inputs with high computational efficiency.






§ INTRODUCTION


Single-image super-resolution (SISR) aims to restore a high-resolution (HR) image from one low-resolution (LR) frame, such as those captured from surveillance and mobile cameras.

The generated HR images can improve the performance of the numerous high-level vision tasks, e.g., object detection <cit.> and face recognition <cit.>.

However, image degradation is inevitable during the imaging process, and would consequently harm the following high-level tasks.

Image blur is a common image degradation in real-world scenarios, e.g., camera or object motion blur during the exposure time.

Furthermore, the blur process becomes more complicated due to the depth variations and occlusions around motion boundaries.

Therefore, real-world images often undergo the effects of non-uniform motion blur. 

The problems of SR and deblurring are often dealt separately, as each of them is known to be ill-posed. 

In this work, we aim at addressing the joint problem of generating clear HR images from given blurry LR inputs.

1 shows one blurry LR image that is affected by non-uniform blur. 

As existing SR algorithms <cit.> cannot reduce motion blur well, the resulting HR image is blurry (see 1_sr_input and <ref>).

On the other hand, the state-of-the-art non-uniform deblurring methods <cit.> generate clear images but cannot restore fine details and enlarge the spatial resolution (see 1_db_input and <ref>).



With the advances of deep CNNs, state-of-the-art image super-resolution <cit.> and deblurring <cit.> methods are built on end-to-end deep networks and achieve promising performance.

To jointly solve the image deblurring and super-resolution problems, a straightforward approach is to solve the two problems sequentially, i.e., performing image deblurring followed by super-resolution, or vice versa.

However, there are numerous issues with such approaches. 

First, a simple concatenation of two models is a sub-optimal solution due to error accumulation, i.e., the estimated error of the first model will be propagated and magnified in the second model.

Second, the two-step network does not fully exploit the correlation between the two tasks. 

For example, the feature extraction and image reconstruction steps are performed twice and result in computational redundancy.

As both the training and inference processes are memory and time-consuming, these approaches cannot be applied to resource-constrained applications. 

Several recent methods <cit.> jointly solve the image deblurring and super-resolution problems using end-to-end deep networks.

However, these methods either focus on domain-specific applications, e.g., face and text <cit.> images, or address uniform Gaussian blur <cit.>.

In contrast, we aim to handle natural images degraded by non-uniform motion blur, which is more challenging.

In this work, we propose a Gated Fusion Network (GFN) based on a dual-branch architecture to tackle this complex joint problem.

The proposed network consists of two branches: one branch aims to deblur the LR input image, and the other branch generates a clear HR output image.

In addition, we learn a gate module to adaptively fuse the features from the deblurring and super-resolution branches.

We fuse the two branches on the feature level rather than the intensity level to avoid the error accumulation from imperfect deblurred images.

The fused features are then fed into an image reconstruction module to generate clear HR output images.

Extensive evaluations demonstrate that the proposed model performs favorably against the combination of the state-of-the-art image deblurring and super-resolution methods as well as existing joint models.

The contributions of this work are summarized as follows:



  * To the best of our knowledge, this is the first CNN-based method to jointly solve non-uniform motion deblurring and generic single-image super-resolution.


  * We decouple the joint problem into two sub-tasks for better regularization.

We propose a dual-branch network to extract features for deblurring and super-resolution and learn a gate module to adaptively fuse the features for image reconstruction.


  * The proposed model is efficient and entails low computational cost as most operations are executed in the LR space.

Our model is much faster than the combination of the state-of-the-art SR and image deblurring models while achieving significant performance gain.






§ RELATED WORK



We aim to solve the image super-resolution and non-uniform motion deblurring problems jointly in one algorithm.

Both motion deblurring and image super-resolution are fundamental problems in computer vision.

In this section, we focus our discussion on recent methods for image super-resolution and motion deblurring closest to this work. 

Image super-resolution.

Single image super-resolution (SISR) is an ill-posed problem as there are multiple HR images correspond to the same LR input image.

Conventional approaches learn the LR-to-HR mappings using sparse dictionary <cit.>, random forest <cit.> or self-similarity <cit.>.

In recent years, CNN-based methods <cit.> have demonstrated significant performance improvement against conventional SR approaches.

Several techniques have been adopted, including the recursive learning <cit.>, pixel shuffling <cit.>, and Laplacian pyramid network <cit.>.

In addition, some approaches use the adversarial loss <cit.>, perceptual loss <cit.> and texture loss <cit.> to generate more photo-realistic SR images.

As most SR algorithms assume that the LR images are generated by a simple downsampling kernel, e.g., bicubic kernel, they do not perform well when the input images suffer from unexpected motion blur.

In contrast, the proposed model performs super-resolution on LR blurry images.


Motion deblurring.

Conventional image deblurring approaches <cit.> assume that the blur is uniform and spatially invariant across the entire image.

However, due to the depth variation and object motion, real-world images typically contain non-uniform blur.

Several approaches solve the non-uniform deblurring problem by jointly estimating blur kernels with scene depth <cit.> or segmentation <cit.>.

As the kernel estimation step is computationally expensive, recent methods <cit.> learn deep CNNs to bypass the kernel estimation and efficiently solve the non-uniform deblurring problem.

Orest et al. <cit.> adopt the Wasserstein generative adversarial network (GAN) to generate realistic deblurred images and facilitate the object detection task.

When applying to LR blurry images, the above approaches cannot produce sufficient details and upscale the spatial resolution.

Joint image super-resolution and motion deblurring.

The joint problem is more challenging than the individual problem as the input images are extremely blurry and of low-resolution.

Some approaches <cit.> reconstruct sharp and HR images from a LR and blurry video sequence.

However, these methods rely on the optical flow estimation, which cannot be applied to the case where the input is a single image.

Xu et al. <cit.> train a generative adversarial network to super-resolve blurry face and text images.

As face and text images have relative small visual entropy, i.e., high patch recurrence within an image <cit.>, a small model can restore these category-specific images well.

Our experimental results show that the model of Xu et al. <cit.> does not have sufficient capacity to handle natural images with non-uniform blur.

Zhang et al. <cit.> propose a deep encoder-decoder network (ED-DSRN) for joint image deblurring and super-resolution.

However, they focus on LR images degraded by uniform Gaussian blur.

In contrast, our model can handle complex non-uniform blur and use much fewer parameters to effectively restore clear HR images.




§ GATED FUSION NETWORK



In this section, we describe the architecture design, training loss functions, and implementation details of the proposed GFN for joint image deblurring and super-resolution.





 §.§ Network architecture


Given a blurry LR image L_blur as the input, our goal is to recover a sharp HR image H.

In this work, we consider the case of 4× SR, i.e., the spatial resolution of L_blur is 4× smaller than that of H.

The proposed model is based on the dual-branch architecture <cit.> and consists of four major modules:

(i) a deblurring module G_deblur for extracting deblurring features and predicting a sharp LR image L, 

(ii) an SR feature extraction module G_SRF to extract features for image super-resolution, 

(iii) a gate module G_gate to estimate a weight map for blending the deblurring and super-resolution features, 

and (iv) a reconstruction module G_recon to reconstruct the final HR output image.

An overview of the proposed model is illustrated in 2.



  
Debluring module.

The deblurring module aims to restore a sharp LR image L from the input blurry LR image L_blur.

Different from the encoder-decoder architecture in  <cit.>, we adopt an asymmetric residual encoder-decoder architecture in our deblurring module to enlarge the receptive field.

The encoder consists of three scales, where each scale has six ResBlocks <cit.> followed by a strided convolutional layer to downsample the feature maps by 1/2×.

The decoder has two deconvolutional layers to enlarge the spatial resolution of feature maps.

Finally, we use two additional convolutional layers to reconstruct a sharp LR image L.

We denote the output features of the decoder by ϕ_deblur, which are later fed into the gate module for feature fusion.



  
Super-resolution feature extraction module.


We use eight ResBlocks <cit.> to extract high-dimensional features for image super-resolution.

To maintain the spatial information, we do not use any pooling or strided convolutional layers.

We denote the extracted features as ϕ_SRF.




  
Gate module.

In Figure3, we visualize the responses of ϕ_deblur and ϕ_SRF.

While the SR features contain spatial details of the input image (as shown on the wall of Figure3(b)), the deblurring features have high response on regions of large motion (as shown on the moving person of Figure3(c)).

Thus, the responses of ϕ_deblur and ϕ_SRF complement each other, especially on blurry regions.

Inspired by the fact that gate structures can be used to discover feature importance for multimodal fusion <cit.>, we propose to learn a gate module to adaptively fuse the two features.

Unlike <cit.> that predicts the weight maps of three derived inputs, our gate module enables local and contextual feature fusion from two independent branches, and thus it is expected to selectively merge sharp features ϕ_fusion from ϕ_deblur and ϕ_SRF in this work.

Our gate module, G_gate, takes as input the set of ϕ_deblur, ϕ_SRF, and the LR blurry image L_blur, and generate a pixel-wise weight maps for blending ϕ_deblur and ϕ_SRF:


    ϕ_fusion = G_gate(ϕ_SRF, ϕ_deblur, L_blur)  ⊗ϕ_deblur + ϕ_SRF,


where ⊗ denotes the element-wise multiplication.

The gate module consists of two convolutional layers with the filter size of 3 × 3 and 1 × 1, respectively.



  
Reconstruction module.

The fused features ϕ_fusion from the gate module are fed into eight ResBlocks and two pixel-shuffling layers <cit.> to enlarge the spatial resolution by 4×.

We then use two final convolutional layers to reconstruct an HR output image H.

We note that most of the network operations are performed in the LR feature space.

Therefore, the proposed model entails low computational cost in both training and inference phases.





 §.§ Loss functions


The proposed network generates two output images: a deblurred LR image L and a sharp HR image H.

In our training data, each blurry LR image L_blur has a corresponding ground truth HR image H and a ground truth LR image L, which is the bicubic downsampled image of H.

Therefore, we train our network by jointly optimizing a super-resolution loss and a deblurring loss:

    min ℒ_SR(H, H) + αℒ_deblur(L, L),


where α is a weight to balance the two loss terms.

We use the pixel-wise MSE loss function for both ℒ_SR and ℒ_deblur, and empirically set α = 0.5.




 §.§ Implementation and training details


In the proposed network, the filter size is set as 7 × 7 in the first and the last convolutional layers, 4 × 4 in the deconvolutional layers, and 3 × 3 in all other convolutional layers.

We initialize all learnable parameters from a Gaussian distribution.

We use the leaky rectified linear unit (LReLU) with a negative slope of 0.2 as the activation function.

As suggested in <cit.>, we do not use any batch normalization layers in order to retain the range flexibility of features.

More implementation details of the network can be found in the supplementary due to space limitation.

To facilitate the network training, we adopt a two-step training strategy.

First, we disable the gate module and fuse the deblurring feature and SR feature directly.


We train the deblurring/super-resolution/reconstruction modules from scratch for 60 epochs.

We initialize the learning rate to 1e-4 and multiply by 0.1 for every 30 epochs.

Second, we include the gate module and train the entire network for another 50 epochs.

The learning rate is set to 5e-5 and multiplied by 0.1 for every 25 epochs.

We use a batch size of 16 and augment the training data by random rotation and horizontal flipping.

We use the ADAM solver <cit.> with β_1 = 0.9 and β_2 = 0.999 to optimize our network.

The training process takes about 4 days on an NVIDIA 1080Ti GPU card.

The source code can be found at http://xinyizhang.tech/bmvc2018http://xinyizhang.tech/bmvc2018 .



§ EXPERIMENTAL RESULTS



In this section, we first describe the dataset for training our network, present quantitative and qualitative comparisons with state-of-the-art approaches, and finally analyze and discuss several design choices of the proposed model.





 §.§ Training dataset


We use the GOPRO dataset <cit.> to generate the training data for the joint SR and deblurring problem.

The GOPRO dataset contains 2103 blurry and sharp HR image pairs for training.

To generate more training data, we resize each HR image pair with three random scales within [0.5, 1.0].

We then crop HR image pairs into 256 × 256 patches with a stride of 128 to obtain blurry HR patches H_blur and sharp HR patches H.

Finally, we use the bicubic downsampling to downscale H_blur and H by 4× to generate blurry LR patches L_blur and sharp LR patches L, respectively.

In total, we obtain 10,7584 triplets of {L_blur, L, H} for training (the blurry HR patches H_blur are discarded during training).

We refer the generated dataset as LR-GOPRO in the following content.



 §.§ Comparisons with state-of-the-arts


We compare the proposed GFN with several algorithms, including the state-of-the-art SR methods <cit.>, the joint image deblurring and SR approaches <cit.>, and the combinations of SR algorithms <cit.> and non-uniform deblurring algorithms <cit.>.

For fair comparisons, we re-train the models of SCGAN <cit.>, SRResNet <cit.>, and ED-DSRN <cit.> on our training dataset.


We use the bicubic downsampling to generate blurry LR images from the test set of the GOPRO dataset <cit.> and the Köhler dataset <cit.> for evaluation.

table_results shows the quantitative evaluation in terms of PSNR, SSIM, and average inference time.

The tradeoff between image quality and efficiency is better visualized in psnr_time_params.

The proposed GFN performs favorably against existing methods on both datasets and maintains a low computational cost and execution time.

While the re-trained SCGAN and SRResNet perform better than their pre-trained models, both methods cannot handle the complex non-uniform blur well due to their small model capacity.

The ED-DSRN method uses more network parameters to achieve decent performance.

However, the single-branch architecture of ED-DSRN is less effective than the proposed dual-branch network.






The concatenations of SR <cit.> and deblurring methods <cit.> are typically less effective due to the error accumulation.

We note that the concatenations using the SR-first strategy, i.e., performing SR followed by image deblurring, typically have better performance than that using the DB-first strategy, i.e., performing image deblurring followed by SR.

However, the SR-first strategy entails heavy computational cost as the image deblurring step is performed in the HR image space.

Compared with the best-performing combination of EDSR <cit.> and DeepDeblur <cit.> methods, the proposed GFN is 116× faster and has 78% fewer model parameters.


We present the qualitative results on the LR-GOPRO dataset in visual_results_GOPRO and the results on a real blurry image in visual_results_real.

The methods using the concatenation scheme, e.g., DeepDeblur <cit.> + EDSR <cit.> and EDSR <cit.> + DeepDeblur <cit.>, often introduce undesired artifacts due to the error accumulation problem.

Existing joint SR and deblurring methods <cit.> cannot reduce the non-uniform blur well.

In contrast, the proposed method generates clear HR images with more details.









 §.§ Analysis and discussion



The proposed GFN consists of several key components:

(1) using a deblurring loss for regularization; 

(2) using a deblurring module to extract the deblurring features; 

(3) using a dual-branch architecture instead of a concatenation of SR and deblurring models; 

(4) fusing features in the feature level instead of the intensity level; 

(5) using a gate module to adaptive fuse SR and deblurring features.

The training schemes of these models are presented in the supplementary material.

Here we discuss the performance contribution of these components.


We first train a baseline model by introducing a deblurring loss to the SRResNet.

We then include the deblurring module to the baseline model using two sequential strategies: DB-first (Model-1) and SR-first (Model-2).

In ablation, the deblurring module shows significant performance improvement over the baseline SRResNet model.

The SR-first model achieves better performance but has a slower execution speed.

However, the concatenation of SR and deblurring modules might be sub-optimal due to the error accumulation.

Next, we employ the dual-branch architecture and fuse the features in the intensity level (Model-3) and the feature level (Model-5).

Compared with Model-3, Model-5 achieves 0.5 dB performance improvement and runs faster.

We also train a model without the deblurring loss (Model-4).

Experimental results show that the deblurring loss helps the network converge faster and obtain improved performance (0.14 dB).

Finally, we show that the learned gate module can further boost the performance by 0.2 dB.



§ CONCLUSIONS

In this paper, we propose an efficient end-to-end network to recover sharp high-resolution images from blurry low-resolution input.

The proposed network consists of two branches to extract blurry and sharp features more effectively.

The extracted features are fused through a gate module, and are then used to reconstruct the final results.

The network design decouples the joint problem into two restoration tasks, and enables efficient training and inference.

Extensive evaluations show that the proposed algorithm performs favorably against the state-of-the-art methods in terms of visual quality and runtime.



§ ACKNOWLEDGEMENT

This work is partially supported by National Science and Technology Major Project (No. 2018ZX01008103), NSF CARRER (No.1149783), and gifts from Adobe and Nvidia.




















