Deep Blind Image Inpainting
Yang Liu Jinshan Pan Zhixun Su
cs.CV







Deep Blind Image Inpainting
    Yang Liu^1, Jinshan Pan^2  , Zhixun Su^1

^1School of Mathematical Sciences, Dalian University of Technology

^2School of Computer Science and Engineering, Nanjing University of Science and Technology














    December 30, 2023
==========================================================================================================================================================================================================================






Image inpainting is a challenging problem as it needs to fill the information of the corrupted regions.
Most of the existing inpainting algorithms assume that the positions of the corrupted regions are known.

Different from the existing methods that usually make some assumptions on the corrupted regions, we present an efficient blind image inpainting algorithm to directly restore a clear image from a corrupted input.



Our algorithm is motivated by the residual learning algorithm which aims to learn the missing information in corrupted regions.

However, directly using existing residual learning algorithms in image restoration does not well solve this problem as little information is available in the corrupted regions.

To solve this problem, we introduce an encoder and decoder architecture to capture more useful information and develop a robust loss function to deal with outliers.

Our algorithm can predict the missing information in the corrupted regions, thus facilitating the clear image restoration.

Both qualitative and quantitative experimental demonstrate that our algorithm can deal with the corrupted regions of arbitrary
shapes and performs favorably against state-of-the-art methods.







§ INTRODUCTION


Image inpainting aims to recover a complete ideal image from the corrupted image.

The recovered region should either be as accurate as the original without disturbing uncorrupted data or visually seamlessly merged into the surrounding neighborhood such that the reconstruction result is as realistic as possible.

The applications of image inpainting are numerous, ranging from the restoration of damaged images, videos, and photographs to the removal of selected regions.
Image inpainting problem is usually modeled as

    x(i) = {[ y(i) + n(i),    M(i) = 1,;        u(i),    M(i) = 0. ].

where x, y and n denote the corrupted image, clear image, and noise, respectively; i denotes the image index; u denotes the values of all other image pixels corrupted by other factors;
M is a binary indicator, in which 0 denotes missing region and 1 indicates that the data is intact (which is usually influenced by noise n).


Recovering clear image y from x is a highly ill-posed problem.
Most existing inpainting algorithms <cit.> usually assume that the corrupted regions are known in advance.
The success of these algorithms is mainly due to the use of different image priors <cit.>.

However, the corrupted regions are usually unknown in most cases, e.g., removing certain scratches from archived photographs.

Most existing methods cannot solve this problem well. 


Although several methods using generative adversarial nets (GANs) <cit.> to solve face image inpainting problems. However, these methods cannot be applied to blind natural image inpainting as these methods highly depend on the face domain knowledge.

Thus, it is great of interest to develop an algorithm to solve the natural image inpainting when positions of corrupted regions are unknown.






In this paper, we propose an efficient blind image inpainting algorithm to directly restore a clear image from a corrupted input using a deep convolutional neural network.

Motivated by the success of the deep residual learning algorithm <cit.>, our deep feed-forward neural network learns the information that is lost in the corrupted regions.

However, as most information is missing in the corrupted regions, directly using the neural networks in image restoration, e.g., <cit.> does not generate clear images.

To solve this problem, we develop an encoder and decoder architecture to capture more useful information. With this architecture, our algorithm is able to predict the missing information in the corrupted regions.

With the learned information, we plug it into the input image and get the final clear image.









The main contributions of this work are summarized as follows.

  
  * We propose an effective blind image inpainting algorithm based on a deep CNN which contains an encoder and decoder architecture to capture more useful information. The proposed deep CNN is based on a residual learning algorithm which is to restore the missing information in the corrupted regions.
  
  * To help the restoration of missing information, we use image gradients computed from the inputs in the residual learning.
  
  * To deal with outlier, we develop a robust loss function, which ensures the training stage is stable.
  
  * The proposed network is trained in an end-to-end fashion, which can handle large corrupted regions and performs favorably against state-of-the-art algorithms.





§ RELATED WORK


Recent years have witnessed significant advances in image inpainting, ranging from diffusion-based algorithms and methods based on statistical priors on natural images to the striking neural network-based algorithms. We briefly review the most representative works related to ours and the applications of the convolutional neural network on image restoration.

Image inpainting. Image inpainting is first introduced in <cit.>, which exploits a diffusion function to propagate low-level features from surrounding known regions to unknown regions along the boundaries of given mask. This work pioneered the idea of filling missing parts of an image using the information from surrounding regions and has been further developed by introducing the Navier-Stokes equations in another work <cit.>. The normalized weighted sum of neighboring known pixels is used to infer the corrupted pixel along the image gradient in a later work <cit.>. In <cit.>, the entire image is first decomposed into the sum of a structure part and a texture part, then these two parts are reconstructed separately based on <cit.>. While these works perform well on image inpainting, global information has not been considered, and they are limited to process small objects.


Numerous image priors have been proposed to tackle image inpainting problem. In <cit.>, hand-crafted features are introduced to help to learn image statistics for inpainting. Roth and Black <cit.> develop the Fields of Experts framework based on a Markov Random Field (MRF) for learning generic image priors which encode the statistics of the entire image and could apply to general texts removal. In another work <cit.>, patch-based Gaussian Mixture prior is proposed based on the EPLL framework which shows good performance in restoring the corrupted region. Barnes et al. <cit.> propose an efficient patch matching algorithm so the user-specified region could be filled in. These priors are powerful and present strong effectiveness in helping inpainting image, yet a problem arises, since they are designed in terms of specific regularization which itself is difficult to develop, it would be more difficult and expensive to utilize prior knowledge from various domains comprehensively.



CNN in image restoration. Recently, we have witnessed excellent progress in image restoration due to the use of CNN.

In image restoration problem, CNN can be regarded as a mapping function, which maps the corrupted images into the clear images. The main advantage of CNN is that it can be learned by large paired training data. Thus, the restoration methods using CNN achieve better performance, such as image denoising <cit.>, super-resolution <cit.> and deblurring <cit.>.

One representative method is the SRCNN which is proposed by Dong et al. <cit.>. This method proposes a three-layer CNN to reconstruct a high-resolution image from a low-resolution input.
Motivated by the success of the residual learning algorithm, Kim et al. <cit.> develop a much deeper CNN model with twenty layers based on residual learning which can generate the results with fine details.






Deep learning based algorithms have been made remarkable progress in image inpainting. Xie et al. <cit.> adopt a deep neural network pre-trained with sparse denoising auto-encoder and demonstrate its ability in blind image inpainting. Ren et al. <cit.> learn a three-layer convolutional neural network for non-blind image inpainting and achieve promising results given the masked image. However, this method assumes that the corrupted regions are known and it cannot be applied to the blind image inpainting problem.


Recently GANs <cit.> has been introduced into face completion. As the methods <cit.> highly depend on the face domain knowledge, they cannot be directly extended to natural image inpainting. In addition, these approaches usually involve introducing the mask image. 

Different from existing methods, we develop a CNN based on the residual learning algorithm to solve blind image inpainting problem. Our proposed network is based on an encoder and decoder architecture, which can preserve the details and structures of the restored images.








§ PROPOSED ALGORITHM


Our algorithm is based on the residual learning algorithm. We develop a deep CNN which contains an encoder and decoder architecture to predict the missing information from the corrupted regions.

In this section, we describe the design methodology of the proposed method, including the network architecture, gradient prior, and loss functions.
























 §.§ Network Architecture


As shown in Figure <ref>, we design a very deep convolutional neural network concatenated by two sub-networks to learn complementary priors and further to learn the underlying mapping.

Given a single masked image to inpainting, firstly the sub-network Net-D takes it as input and outputs a multi-channel feature map which captures complementary image priors. Then these learned priors and pre-extracted gradient prior are further copied and concatenated with the masked image. After that, the sub-network Net-E takes it as input and aims to learn the underlying residual mapping <cit.>. Finally, the target-like restored image is yielded by pixel-wisely summing the input masked image and output residual image.

We note that the learning process of sub-network Net-D is a decoder which captures high dimension complementary image priors from low dimension input image. On the contrary, the sub-network Net-E performs the inverse transformation of Net-D which encodes the low dimension corrupted data in terms of the high dimension image information. Under these observations, we denote these two sub-networks by Net-D and Net-E.

The proposed network has 40 layers.

Net-D contains 17 convolution layers and 3 de-convolution layers, while Net-E contains 20 convolution layers. The filter size of theirs is all 3 × 3, each followed by a rectified linear unit(ReLU).

Inspired by the philosophy of VGG nets <cit.> and VDSR <cit.>, the filter number of the first layer is 64 which remains unchanged for the same output feature map size and is halved/doubled for the max pooling/de-convolution operation.

Net-D has 3 additional max pooling operations with stride 2 of size 2 × 2 adopted for downsampling. Besides, 3 skip link operations are used to copy and concatenate the feature maps, inspired by U-Net <cit.>, which has been confirmed to favor propagating low-level context information to higher semantic layers.

We apply the zero padding of size 1 to keep all of the feature maps the same size. In this way, the residual image of the network is directly added back to the input image pixel-wisely to yield the final output image. We use this network architecture for all experiments in our work.



 §.§ Pre-extracted Gradient Prior 



We note that image gradients model the structures. Thus, we further use image gradients to help the estimation of the details and structures in the corrupted regions.


Given the corrupted image, we extract its horizontal and vertical gradients. 

These two directional gradients are then copied and concatenated with the input image to help the detail and structure estimation.

We will demonstrate the effect of the gradients in Section <ref>.





 §.§ Loss Function


Given a training dataset {x^(i), y^(i)}_i=1^N, a natural way to train the proposed network is to minimize the loss function

    L(x, y) = 1/N∑_i=1^Nϕ ((f(x^(i)) - y^(i))

where f is the mapping function which is learned by the proposed network, and ϕ(·) denotes a robust function.

In this work, we take ϕ(·) to be the L_1 norm instead of the L_2 norm to increase the robustness to the outliers.
As the network proposed is based on the residual learning algorithm <cit.>, the loss function used for the training is:

    L(x, y) = 1/N∑_i=1^N‖(f(x^(i)) + x^(i) - y^(i)‖_1

We will show that the loss function based on L_1 norm is more robust to outliers thereby leading to higher accuracy and visual quality in Section <ref>.





§ EXPERIMENTAL RESULTS


In this section, we evaluate our algorithm on several benchmark datasets and compare it with state-of-the-art methods. 

Similar to existing methods <cit.>, we only train our model on grayscale images, and all the PSNR and SSIM values in this paper are computed by grayscale images. 

We use color images for visualization, where each channel of the color image is processed independently. 

The trained models and source code will be made available to the public.







 §.§ Datasets

We create a binary mask image dataset which contains 20 text images for generating training dataset and another 10 text images for generating test dataset with image size of 512 × 512. Randomly distributed squares with the size of 5 to 40 pixels are added.

The texts are of the most popular font Arial and Helvetica Neue with different font sizes of 10 to 50 pt and the most common font styles, including bold, thin, italic, light, ultralight, underline, condensed styles and their combinations. Note that the texts and squares indicate the corrupted regions which are unknown in advance in blind inpainting task.

Training dataset. We generate the ground truth of the training data by randomly sampled 100,000 64 × 64 grayscale patches from 291 natural images <cit.> with data augmentation, e.g., scale and rotation. Then the masked patches are generated by taking a pixel-wise multiplication between the ground truth and the created 20 text images for training, plus randomly distributed squares.

Test dataset. Four datasets are employed for evaluating the proposed algorithm, i.e., Set5, Set14, Urban100 <cit.> and BSDS500 <cit.>, which have been widely used as benchmark test datasets in image restoration problems. The masked test images are generated by taking the same pixel-wise multiplication with the created 10 text images for test, plus randomly distributed squares which are identical for each test method.



 §.§ Network Training


Our implementation is based on PyTorch. We trained all networks on an NVIDIA 1080 Ti GPU using the created 100,000 image patches of the size 64 × 64 pixels.

The weights of our proposed network are initialized by the method proposed by He et al. <cit.>, which is confirmed to favor the performance of the rectified linear unit (ReLU).

We take Adam <cit.> as the optimization method with β = (0.9, 0.999) and set the weight decay to be 0. The number of the optimal training epoch is 60 (47100 iterations in total with batch size 128). The network is first trained with 50 epochs at a learning rate of 10^-3, then another 10 epochs at a lower rate of 10^-4. Training with too many iterations has no improvement for the results in terms of our observations.







 §.§ Comparisons with State-of-the-Art Methods


We present experimental evaluations of the proposed algorithm against several state-of-the-art methods, including blind inpainting methods and even non-blind inpainting methods <cit.>, in terms of their publicly available codes. Table <ref> summarizes the quantitative evaluations on several benchmark datasets, and our method performs favorably against previous methods even non-blind methods by a large margin under the measurements PSNR and SSIM.

Running time. We benchmark the running time of all methods on dataset Set5 (by grayscale) with a machine of an Intel Core i7-6850K CPU and an NVIDIA GTX 1080 Ti GPU. Table <ref> shows that our method runs much faster than other methods. Moreover, it just costs 0.007 seconds to test a grayscale image of size 512 × 512, e.g., the image baby_GT.bmp in Set5, making our model capable of inpainting image in real-time.

Visual Comparisons. Figure <ref> shows several visual comparisons with state-of-the-art methods. Due to the intrinsic difficulty and challenge, previous blind and non-blind methods fail to recover missing information of square regions. Moreover, these methods still contain significant artifacts in dealing with texts. In contrast, our method generates much clearer images. Figure <ref> shows another set of comparisons tested on an old photograph. This is a real image from <cit.>. Our method could generate considerably realistic results.






§ ANALYSIS AND DISCUSSION


In this section, firstly we further examine the effect of different parts of our proposed network via qualitative and quantitative experiments. 
All of them are conducted in terms of the same parameters described in Section <ref>. 
Then we analyze convergence properties of our networks.
Finally, we discuss the limitations of the proposed method.



For simplicity and clarity, we denote our complete network and method as Net-D+GP+Net-E+L_1, where GP is short for pre-extracted gradient prior. 

The network GP+Net-E+L_1 denotes training without Net-D, and the network Net-E^64+GP+Net-E+L_1 denotes replacing Net-D with Net-E^64, where Net-E^64 is a 20-layer plain network that has the same architectures to those of Net-E except that its last layer has 64 channels. 

Another two networks Net-D+Net-E+L_1 and Net-D+GP+Net-E+L_2 denotes training without gradient prior and training using L_2 loss function, respectively.


















 §.§ Effect of Net-D





We examine the effect of Net-D by replacing it with different sub-networks and compare with GP+Net-E+L_1 and Net-E^64+GP+Net-E+L_1.

As mentioned in Section <ref>, Net-D is used to capture more useful information and it plays an important role for the inpainting.

Table <ref> shows the quantitative comparison between Net-D+GP+Net-E+L_1 and GP+Net-E+L_1. We can see that the PSNR/SSIM of training with Net-D is significantly higher than that of training without Net-D. A visual comparison is presented in Figure <ref>. Net-D+GP+Net-E+L_1 can recover sharp edges while the results of GP+Net-E+L_1 contains some artifacts.








We note that the skip connection operations <cit.> play a critical role in Net-D, which are used to copy and concatenate shallow feature maps with deep feature maps. In this way, Net-D could output comprehensive image priors combining low-level features e.g., edges and boundaries, with high-level features e.g., contours and objects.

We compare with Net-E^64+GP+Net-E+L_1, which is a plain network without any skip connection. This network aims to learn high-level semantic features. It is surprising that this network even does not achieve better quantitative evaluations than GP+Net-E+L_1. Although Net-E^64+GP+Net-E+L_1 can still remove texts well, it fails to recover sharp edge as better as Net-E+GP+Net-E+L_1 when dealing with large squares, suggesting the effectiveness of Net-D.



 §.§ Effect of Pre-extracted Gradient












The priors learned by Net-D and pre-extracted gradient prior are both employed in our network as showed in Figure <ref>. 



We use the pre-extract gradients to complement and enhance the features learned by Net-D. 

We train a network without pre-extracted gradient prior named Net-D+Net-E+L_1.

Table <ref> summarizes the quantitative evaluations of these two networks in terms of PSNR and SSIM, and Net-D+GP+Net-E+L_1 achieves higher results than Net-D+Net-E+L_1 on the benchmark datasets.

Figure <ref> shows some visual comparisons of using pre-extracted gradient or not. When gradient has been used, the network Net-D+GP+Net-E+L_1 is more likely to recover the main structures and generate sharper edges (Figure <ref>(c)), while the network Net-D+Net-E+L_1 trained without gradient is less accurate (Figure <ref>(d)), suggesting the effectiveness of pre-extracted gradient.





 §.§ Effect of the Loss Function







We note that L_2 norm based loss function is widely used in CNN to restore image<cit.>.
However, As most information is missing in the corrupted regions, e.g., the tagged region in Figure <ref>(a), lots of outliers may be introduced. 
Thus the generated image obtained by L_2 loss function contains significant artifacts (Figure <ref>(d)).


To examine the proposed loss function in image inpainting, we train a network Net-D+GP+Net-E+L_2 using the L_2 loss function. 
Figure <ref>(d) shows a recovered image. We can see that the resulting image contains undesirable artifacts and the learned grayscale residual image (the second close-up in Figure <ref>(d)) is significantly different from the ground truth.

Under these observations, L_1 loss function as depicted in (<ref>) is adopted in this work to alleviate the effect of outliers. As a contrast, the restored image in Figure <ref>(c) is sharper, and the resulting grayscale residual image gets much closer to the ground truth.
Figure <ref> shows the values of the training loss functions based on L_1 norm and L_2 norm. The method with the L_1-norm-based reconstruction error converges better than that with the L_2-norm-based reconstruction error, both in training and test stage, suggesting that L_1 loss function is more robust.

Quantitative evaluations for L_1 and L_2-norm-based loss function on four benchmark test datasets are summarized in Table <ref>.
Net-D+GP+Net-E+L_1 has higher PSNR and SSIM values than those of Net-D+GP+Net-E+L_2.



 §.§ Convergence Property

We quantitatively evaluate the convergence properties of our method against the other four networks, i.e., GP+Net-E+L_1, Net-E^64+GP+Net-E+L_1, Net-D+Net-E+L_1 and Net-D+GP+Net-E+L_2, on the dataset Set14. Figure <ref> shows the convergence rate on the dataset Set14.

The proposed algorithms converge well after 50 epochs.



 §.§ Limitations




The proposed algorithm directly restore a clear image from a corrupted input without any assumptions on the corrupted regions.

However, it does not work well when important structures or details are corrupted. These structures or details are usually unique in an image. 

Figure <ref> shows a failure example, where the nose and mouse are heavily corrupted, our method fails to recover these two parts. 

Future work will consider using an adversarial learning algorithm to solve this problem.








§ CONCLUSION

In this paper, we have proposed an efficient blind image inpainting algorithm that directly restores a clear image from a corrupted input.

We introduce an encoder and decoder architecture to capture useful features and fill semantic information based on the residual learning algorithm. 
We develop L_1 loss function in the proposed work and show that it is more robust to outliers.

Extensive qualitative and quantitative experiments testify to the superiority of the proposed method over state-of-the-art algorithms.



ieee




