Interactive Deep Colorization With Simultaneous Global and Local Inputs
Yi Xiao Peiyao Zhou Yan Zheng
cs.CV





𝐱
ŁL










Yi Xiao^1, Peiyao Zhou^1, Yan Zheng^2

1 College of Computer Science and Electronic Engineering

2 College of Electric and Information Engineering

Hunan University

Changsha, China










Interactive Deep Colorization With Simultaneous Global and Local Inputs
    
Guangwei Wu^
         Jianer Chen^
         Jianxin Wang^[Corresponding author,
        email: jxwang@csu.edu.cn.]



^School of Information Science and Engineering

    Central South University

    ChangSha, Hunan 410083, P.R. China



^College of Computer and Information Engineering

    Central South University of Forestry and Technology

    ChangSha, Hunan 410004, P.R. China



^School of Computer Science & Education Software
 
     Guangzhou University

     Guangzhou, Guangdong 510006, China



^Department of Computer Science and Engineering 

    Texas A&M University

    College Station,  TX 77843, USA

    
=================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================





 Colorization methods using deep neural networks have become a recent trend. However, most of them do not allow user inputs, or only allow limited user inputs (only global inputs or only local inputs), to control the output colorful images. The possible reason is that it's difficult to differentiate the influence of different kind of user inputs in network training. To solve this problem, we present a novel deep colorization method, which allows simultaneous global and local inputs to better control the output colorized images.  The key step is to design an appropriate loss function that can differentiate the influence of input data, global inputs and local inputs. With this design, our method accepts no inputs, or global inputs, or local inputs, or both global and local inputs, which is not supported in previous deep colorization methods. In addition, we propose a global color theme recommendation system to help users determine global inputs. Experimental results shows that our methods can better control the colorized images and generate state-of-art results.

 


     


Colorization, Deep convolution Neural networks, Color theme, User input






§ INTRODUCTION

Image colorization refers to the technique that adds colors to monochrome images or videos.
Generally speaking, colorization is a ill-posed problem, it does not have a unique solution.
To get satisfactory colorized results, two categories of method have been proposed: user-guided edit propagation and data-driven automatic colorization.


The user-guided edit propagation methods <cit.>  require the user to draw colored strokes
and propagate the colors across the image by solving a global optimization problem.
 These methods can achieve impressive colorized images, but often require a very large number of scribbles for images with complex textures. This is because each different color
region must be explicitly marked by a different colored stroke, even regions with obvious semantic hints, such as a blue sky or green trees, need to be specified by the user.

To address this problem, early data-driven colorization methods proposed to automatically colorize a grayscale image by learning the color hints from one or several exemplar color images with similar semantics <cit.>. Unfortunately, it may be time-consuming or hard to find a suitable exemplar image sometimes. With the popularity of deep learning, recent data-driven colorization methods using deep neural networks have become a recent trend <cit.>.
Using a large number of grayscale and color image pairs, the deep colorization methods learn a parametric mappings for fully automatic colorization. These methods can generate plausible colorful images in most of time. However, since an semantic region can have multiple choices of colors, the results can contain colors or styles which users do not expected. For example, users may want a green mountain in spring, but get a yellow mountain in autumn.

The excellent recent work, user-guide deep colorization,  by Zhang et al <cit.> combines the advantages of user-guided and data-driven methods. It provide better user controls by taking either global inputs or local inputs in the deep network training. The color of the colorized image can be controlled by a global color histogram, or a few local color points. The user-guide deep colorization <cit.> can generate plausible colorized images according to the users expectation with only a few inputs. However, The user-guide deep colorization method does not support simultaneous global inputs and local inputs, it can only allow one kind of inputs at one time.

We argue that supporting simultaneous global inputs and local inputs can provide better control on the output images. The ideal case is that a user can control the overall color style of the image with global color inputs, and meanwhile assign local colors to certain regions with local inputs.

Supporting multiple kinds of inputs simultaneously in deep networks is not straightforward. It's difficult to differentiate the influence of different kind of user inputs in network training. An example is shown in Figure <ref>, the influence of local inputs is lost if the loss function is not designed appropriately.  To solve this problem, we present a novel deep colorization method, which allows simultaneous global and local inputs to better control the output colorized images. Our basic model neural network is a U-net network including three parts, the feature extraction part, the fusion part, and the reconstruction part. The feature extraction part, consisting of 3 convolution layers and 3 pooling layers, extracts features from the gray-scale images and the local input images. The fusion part fuse the global inputs (color themes) and the extracted feature maps. The reconstruction part, consisting 3 deconvolution layers, then reconstructs the two chrominance channels (Lab space). The key step is to design an appropriate loss function that can differentiate the influence of input data, global inputs and local inputs. With this design, our method accepts no inputs, or global inputs, or local inputs, or both global and local inputs, which is not supported in previous deep colorization methods.

Different with previous works, which use exemplar images or histograms as global inputs, we choose to use the color theme, a template of colors possibly associated verbal description <cit.>, as the global input. It is more easier for users to assign a color theme than to choose exemplar images or histograms. In our model, we accept color themes consist of 3 - 7 colors. More colors are possible but difficult to use. To further save work time for users, we propose a global color theme recommendation system to help users determine the global input. Inspired by <cit.>, we build a mapping between grayscale texture features and color histograms from a large image data base, and use the mapping to predict the color themes. Also, the user can define its own color theme.

Experimental results show that the color images generated by our method look real and natural, and the detail of the image are preserved well. Using different color themes and local inputs, we can output different style images (See Figure <ref>). To summarize, our methods can better control the colorized images and generate state-of-art results.

Our contributions in this paper are as follows:


  * We propose a novel deep colorization methods that supports no inputs, or global inputs, or local inputs, or both global and local inputs.

  * Our methods enables global inputs using color themes of variable color numbers.

  * We propose a color theme recommendation system which can suggest the color themes for users.


























 













§ RELATED WORK



Image colorization is always a hot topic in the filed of image process. After the advent of deep convolution neural network, it is devoted to train images on network to achieve the goal of great colorization. At present, there are two main ways of colorization. One is user-guided edit propagation colorization, and the other is automatic colorization.


About user-guided coloring method, most of them are devoted to add scribbles on gray images. In the early paper, like <cit.>, it is necessary for users to draw some desired color graffitis on certain regions, through the color propagating among pixels which are similar in intension to achieve the effect of colorization. <cit.> and <cit.> worked on the issue of edge detection and enhancement, which can improve the quality of color transmission. In  <cit.>, users select the area of interest and then specify some color points within the area.  <cit.> and  <cit.> are optimized for edit propagation and shading. It’s also popular to use color theme, such as using data driven color theme enhancements in  <cit.>, which is used to change the color style of color images through different color themes. We take this method and using color theme as global input to color gray images.   <cit.> is a popular method to use the palette to change the corresponding color on color image as same as  <cit.>. In <cit.>, it is proposed to automatically learn the degree of similarity between user strokes and input images to propagate graffiti color. <cit.> uses generative adversarial networks to achieve transforming sketch to real images, which also supports to add user strokes. In <cit.>, by adding global features and local features on deep convolution neural network to train a colorization model, separately, users can specify the color at any location to render the color, or use a reference image to transfer global color. Our method can achieve colorization using both of them simultaneously. There are online applications developed mainly for line-drawing colorization, like  <cit.> and  <cit.>.

For more convenient to color grayscale images globally, there are some method using reference images as global inputs early. <cit.> transfers colors between images with a simple algorithm. <cit.> and <cit.> transmit the color emotion of reference images to grayscale images by matching the brightness and texture information between images. <cit.> needs users to provide the semantic information of the foreground image. The system searches for and downloads pictures with the same foreground semantics on the network, and use them to color the foreground and the background. <cit.> can quickly transfer the color of a reference image to the target image by using a fast cascade feature matching scheme to exploit multiple image features.

But using reference images which is similar with target images to render the color needs a large image database. Users also have to spend a lot of time to choose satisfactory reference image. So, the research of automatic colorization gets more and more attention. <cit.> uses algorithm to reserve grayscale images or videos. In this method, some representative color points are reserved and used to restore these medias. <cit.> uses machine learning tools to extract as much information as possible from the color sample dataset and then estimates the probability color distribution for every pixel. <cit.> matches grayscale image with reference images downloaded automatically from web to transfer color. <cit.> train the objective function based on image features on the LEARCH framework and achieve the coloring effect by minimizing the objective function. <cit.> introduces the concept of feature descriptor, and take the feature descriptors extracted from the grayscale image as input, and finally output the corresponding color values of the UV channel. In <cit.>, a convolutional neural network (CNN) which accepts black and white images as input is designed and constructed, and a statistical learning driven method is used to solve the problem of grayscale colorization. The end-to-end network in <cit.> adds global feature and uses classification label to optimize output results. In addition, <cit.>, <cit.>, <cit.>,  <cit.>, <cit.> also achieve the effect in coloring gray image without users' intervention. <cit.> uses conditional adversarial networks and <cit.> proposes a setup utilizing two networks in tandem to achieve colorization, which both are worth thinking about.

Neural network also has shown us some surprising results in other fields of image processing.  <cit.> reaches the target of automatic style transferring between images. As well as large-scale image recognition  <cit.>,  automatic photo adjustment  <cit.>, sketch simplification  <cit.>, context encoders  <cit.> and cartoon colorization  <cit.>. Accuracy and practicability of neural network is the reason that we chose neural network to research and construct satisfactory colorization method.








§ OUR METHOD






Our deep network model takes in a gray-scale image (L channel in CIE Lab space), a global input and a local input and output the corresponding ab channels. The structure of the model is shown in Figure <ref>. The global input includes a color theme of several colors and a mask indicating the number of colors (all black means no global inputs). The local input includes a image consisting of the assigned local color points and a mask indicating the position of color points (all black means no local inputs). Therefore, by setting the masks of global inputs and local inputs, our model accepts four combination of user inputs: no inputs, global inputs, local inputs, global inputs and local inputs.













 §.§ Problem Formulation

The inputs of our model are a grayscale image X∈ R ^ H× W× 1,
 a global input tensor U_g∈ R^1 × K × 3, and a local input tensor U_l ∈ R^H× W× 3, where H, W is the width and height of the input image, and K is the number of colors in the color theme. The output is a tensor O ∈ R^ H× W× 2.
We aims to train a convolution neural network (CNN), denoted by ℱ(X,U_g,U_l;θ), to approximate the mapping between gray and color, under the constraint of user inputs. Therefore, the colorization problem can be formulated as


    θ^* = min_θ𝔼_X,U_g,U_l,Y,𝒟[ ℒ(ℱ(X,U_g,U_l;θ),U_g, U_l, Y)],


where 𝒟 denotes the training data set, ℒ denotes the loss function, and Y ∈ R^ H× W× 2 is ground truth image color. More details of the loss function will be presented in Section <ref>.
It should be noticed that we train a single model to handle simultaneous global input and local input. Moreover, our loss function is explicitly related to the global input and the local input. These are the two key points which differ our method from <cit.>.





 §.§ User inputs



Global inputs:
To control the overall color style of the output, we design the global input as a color theme with K colors plus a one-channel mask indicating the number of colors. An example of color theme with 5 colors and the corresponding mask are shown in Figure <ref>. To prepare training data for the global input, we use the K-mean clustering algorithm to find the K representative colors for each color image in the data set. The K representative colors form the global color theme. To enable color theme with variable number of colors, K randomly varies in the interval [3,7]. Color theme with more colors are also supported by modifying the interval, but may be harder to use. The ab channels of each color theme U_g^c∈ R^1 × K × 2 and its mask  M_g∈ R^1 × K × 1 forms the global input U_g = {U_g^c, M_g}∈ R^1 × K × 3





For each color image, we generate a K-color map by decoding the color image with its representative colors. Figure <ref> shows an example. The ab channels of K-color map, I∈ R^H × W × 2 is used to calculate the loss function which measures the similarity between the color theme and the output image. More details of loss functions will be described in Section <ref>.
















Local inputs:
Similar to <cit.>, we prepare the training data for the local inputs colors U_l^c∈ R ^ H× W× 2 by randomly selecting some points from the ab channels of every color image. If there are no samples in certain position, the ab values are set to zeros. A one-channel mask M_l ∈ R ^ H× W× 1 is also generated to indicate the positions of the local inputs.
"1" in certain position means there is an input there, while "0" means there are no inputs in the position. Finally, the local inputs U_l = {U_l^c, M_l}∈ R ^ H× W× 3.





 §.§ NetWork model

As shown in Figure <ref>, our network model uses a U-Net structure <cit.>, which is already shown to works well in colorization <cit.>.
It is mainly composed of four parts: the feature extraction module, the global input module, the fusion module and the reconstruction module.





  §.§.§ Feature extraction module

The feature extraction module corresponds to the blue part in Figure <ref>.
The feature extraction module takes in a grayscale image X∈ R ^ H× W× 1 and a local input U_l∈ R ^ H× W× 3. Initially, X and U_l are respectively convolved to a tensor of size H× W× 32. These two tensors are merged to a single tensor of size H× W× 32 by linear interpolation, whose weight is treated as a parameter of the network to be trained. The merged tensor is then feeded to the following convolution blocks.

In convolution block 2 to 7, all the convolution kernels are  3× 3. Through every layer, only the tensor sizes of input are halved spatially with the stride 2× 2, or only the tensor dimensions are doubled with the stride 1× 1. After conv7, the tensor is H / 8 × W / 8 × 512. We use convolution layers with stride 2× 2 instead of using max-pooling layers to reduce the tensor sizes, which is helpful in increasing the spatial support of each layer <cit.>.  Conv8 and conv9 process the tensor further with convolution layers to reduce the 512-channel tensors to 256-channel tensor. The output of feature extraction module is a tensor of size H / 8 × W / 8 × 256, which will be fused with the global input.

 























  §.§.§ Global input module

The global input module, the pink part in Figure <ref>, is one of the characteristics of our method. It takes in the global inputs U_g ∈ R ^ 1× K × 3, which consists of the ab channels of the color theme U_g^c and the corresponding mask M_g. To to unify its size with the size of the fusion module, we reshape it to a tensor of size 1×1× 3K, which is then processed by three fully connected layers. This operation is similar to the global feature network in <cit.>, but we use it to handle user global inputs instead of the features exacted from the input gray-scale image.





  §.§.§ Fusion module

The fusion layer, shown with yellow color in Figure <ref>, is an important step to fuse the global input and the extracted features. Similar operations are also used in <cit.>. The global input is fused with the output of the feature extraction module by linear interpolation, whose weight is also treated as a parameter in the network to be trained. Finally the fusion module outputs a feature tensor of size H / 8 × W / 8 × 256 .




  §.§.§ Reconstruction module


After the fusion module, the feature tensor is processed by a set of convolution layers and upsampling layers for ab channel reconstruction, shown with purple color in Figure <ref>. Convolution layers reduce tensor dimensions by half and upsampling layers doubles the wide and height of the tensor. Conv17 is the last convolution layer with a Sigmoid transfer function, followed by an upsampling layer. The advantage of Sigmoid is that the output range is limited. The output tensor is of size H× W × 2, which combines the input gray-scale image
 X∈ R ^H× W× 1 to generate the final output O ∈ R ^ H× W × 3.






 
 
 











 §.§ Loss function


Designing an appropriate loss function is the most important part of our work. It is not straightforward to
design a loss function which can differentiate the impact of different inputs. We interpret the four combinations of inputs as follows:


  1 No inputs means the user wants the colors to be assigned with the “experiences" of training data.

  2 Only the global input means the user wants the colors to be conditionally assigned with the "experience" of training data.

  3 Only the local input means the user wants to assign his/her preferred color to certain region, but let the colors of other regions be assigned with the “experiences" of training data.

  4 Both inputs mean that the user wants to assign his/her preferred color to certain region, but let the colors of other regions be conditionally assigned with the "experiences" of training data.
















  §.§.§ Loss function for no input or global input


For no inputs or only the global input, the straightforward choice is to measure the differences between ab channels of the output image O∈ R ^ H× W × 2 and the ground truth color Y∈ R ^ H× W × 2. There are many kind of loss functions as discussed in <cit.>. We choose the Huber loss as it produces relative high saturation effect. The Huber loss is given by

    ℒ_H(O,Y) = {[ 1/2(O - Y)^2  | O - Y| ≤δ;                          ;     δ| O - Y| - 1/2δ ^2 , ].

where δ is the parameter of the Huber loss. The value of δ will slightly affect the results as shown in Figure <ref>. We set δ = 0.5, as this value works well in our experiments.










However, in some cases, the impact of the global input (color theme ) is not so obvious. Therefore, we add an additional part which calculates the Huber loss of the output image and the K-color map I, which is defined by decoding the ground truth with the color theme (see Section <ref> and Figure <ref>).

Therefore, the loss function for this part is given by

    ℒ_g  = α_1*ℒ_H(O,Y) + α_2*ℒ_H(O,I),

where α_1 and α_2 are two parameters to balance the influence of two parts. The influence of the two parameters is shown in Figure <ref>. As we expect that output images not only keep natural color but also reflect colors of the color theme, we set α_1=0.7 and α_2=0.3, with which the output can preserve the characteristics of the color theme with almost no color overflows, as showed in Figure <ref>.
When there is no color theme input, the K-color map image will be replaced by the ground truth. In this case, ℒ_g = ℒ_y.
































  §.§.§ Loss function for local input

The local input in our method is U_l = {U_l^c, M_l}∈ R ^ H× W× 3, which is combined by the ab channels of the user input and the corresponding mask, as introduced in Section <ref>. If a user provides a local input in a certain position, we think the user wants the preferred colors instead of colors assigned by the “experience" of training data. In there is only the local input, the Hubber loss ℒ_H(O,Y) defined in Equation (<ref>) is already shown to works well <cit.>, since the differences of the output image and the local inputs at positions with local inputs are already contained in Equation (<ref>).

However, if there are simultaneous global input and local input, the case becomes more complex.
The Hubber loss may seems to work sometimes, when the local inputs are consistent or close with the global color theme, as shown in Figure <ref>(f). For example, the input brown color in the back and swing is close to one color in the color theme (Figure <ref>(a)). In this case, the local input can enhance the colors of local regions compared to results with only global inputs (Figure <ref>(b)). But there exist some colors which overflow from the bird body to the background (TODO, indicate in the figure).

The color overflow can be removed by adding the gradient of the output image and the ground truth in the loss function, given by

    ℒ_ s =(  O _ Sobel ,  Y _ Sobel ),

 where MSE denotes the mean squared error, Sobel denotes the Sobel gradient operator. As shown in Figure <ref>(g), the color overflows are removed at the cost of removing the local color impacts on the head and abdomen.

When the local input colors are quite different with the colors in the global color theme, the phenomenon is more obvious, as shown in Figure <ref>(j)(k). Using only the loss L_g cannot guarantee the local color impact, and results in more obvious color overflows (Figure <ref>(j)). The gradient loss ℒ_ s almost remove the color overflows, but also remove the impact of local color inputs (Figure <ref>(k)).
Since we expect the colorized image preserves not only the global input color theme, but also the local input colors, without few color overflows. We emphasize the impact the local inputs by adding the loss of the output image and the local inputs at positions with local inputs, given by

    ℒ_ p =MSE( ( O∗ M _ l ), (  U _ l ^ c ∗ M _ l )  )

where MSE denotes the mean squared error, M_l is the mask of the local input. With ℒ_p, we can see from Figure <ref>(h)(i) that the aforementioned problems are solved.





























To summarize, our final loss function ℒ is

    ℒ(ℱ(X,U_g,U_l;θ),U_g, U_l, Y) 
    
       =ℒ_ g +ℒ_ s +ℒ_p.


















 §.§ Color theme recommendation system



To help user determine the possible color theme of a gray-scale image, we propose a color theme recommending system.
Our system is inspired by the data driven method in <cit.>, which consists of an off-line process to builds a texture-color distribution mapping and an on-line process to lookup the color distribution for the segmented input color image. 
Different with the off-line process in <cit.>, which built its texture-color distribution mapping on color images to constrain the color distribution of textures, we build the mapping on the gray-color image pairs to predict the colors themes.









 



The framework of color theme commended system is illustrated in Figure <ref>. It consists of an off-line phase(lower row) and an online phase(upper row). The mapping of texture-color distribution is obtained in the off-line stage. The color themes of gray-scale image that input by users can be predicted by computing the main texture of gray-scale image in online phase.
 

During the off-line phase, we start with building an image database, by randomly choosing thousands of colorful images from Imagenet, and then convert the colorful images to grayscale images to form a grayscale image database. Next we adapt the graph-based method <cit.> to segment each gray-scale image into segmentations. We use the mean and standard deviation of the
pixelwise Gabor filter (scale=4, rotation=6) responses within every segmentation, resulting a 24 dimensional texture descriptor for every segmentation. We then build a texture library by clustering all the texture features into 120 textures with k-means algorithm. For each cluster, we accumulate a 2D color distribution with 100 bins by quantizing the corresponding ab channels of the segmentations. Hence, we build a texture-color distribution mapping.

In the online-phase, we segment the input gray-scale image, extract the texture feature vectors, find the closet clusters in the texture library, and lookup the corresponding color distributions. We choose the most frequent color of the K largest segmentations (with more number of pixels) as our recommended color theme.
 We display some images colored by color themes which are generated by this system in Figure <ref>.



















  
  
  
  










§ EXPERIMENT
















  
  
  
  






  
  
  
  







  
  
  
  






  
  
  
  









  
  
  
  





















































  
  
  
  








We implemented the network model on the NIVIDIA GTX1080Ti GPU, using Google's TensorFlow architecture. We use the Place dataset to train our model. Figure <ref> shows some colored results of this dataset. The K-color map images, color themes, local inputs and their corresponding masks are generated on Matlab. Then, we randomly adjust the order of training sets as input of our model. We train the model using a batch size of 50 for 40,000 iterations, which takes about two days on the GPU. To enable the four combinations of input: no inputs, only global input, only local input, and both global and local input, we randomly select one kind of combination in the training. The ratio to select each combination is equal to 25%.



In the previous works, the normalization of input has been shown to speed up the learning speed. In Lab space, the range of L is [0-100], and the range of ab channel is [127, -128]. In order to accelerate the learning speed, we normalize the range of L and ab to [0,1] as

    [               L = L/100; ab = ( ab + 128 ) / 255 ]





































 §.§ Automatic Colorization

In these section, we compare our automatic colorization method with several state-of-art methods, including  <cit.>, which train coloring with no inputs or local inputs ,  <cit.> and  <cit.>, which focus on automatic colorization. The colorized results of these four methods are shown in Figure <ref>. As shown in the figure, all the automatic methods can colorize the gray-scale images. But due to the different training sets, the colorized images may have different color styles. In terms of numerical evaluation, our method and  <cit.> get better PSNRs without compared with the other two methods. All PSNRs of our automatic results are higher than  <cit.>. Note that, although automatic colorization is only our by-product, it still has good performances.




























 §.§ Interactive Colorization



In this section, we compare our interactive colorization method with the state-of-art user guide method <cit.>, and the classical optimization method <cit.>. The results are shown in Figure <ref>. The Butterfly images in the first three columns show results colorized by different local input points. With only local inputs, the results of our method are similar to those of the user-guide method <cit.>. In contrast, the results of <cit.> fail to diffuse the color to the whole butterfly.




For the next five images, our method uses global input (color theme) and local input together.  The other two methods use local inputs only. For images with less clear boundaries, like the fourth column, our method can faithfully assign green to the trees, where the other methods cause color overflows to the background.
Using our method, users can control the colorized image by choosing a color theme, and add a small number of local input points to certain regions, while other methods maybe need to add more local input points to produce the same results. For example, in the fourth column, users maybe need to add several extra color points on the house and sky to avoid color overflow and assign certain colors for the user guide method <cit.>.  <cit.> needs to assign more color points for everywhere in every image. Otherwise, the whole image is going to be one color as shown in the figure. Besides, our method also shows good performance on other type images, like human and natural pictures showed in the last three columns of the Figure <ref>.

In general, our method provides a more convenient and efficient tool for the users by supporting four combinations of input in a single network: no inputs, only global inputs, only local inputs, and simultaneous global input and local input.







 §.§ Numerical Comparisons


Besides visual comparisons, we also evaluate the PSNRs of the related methods. We randomly choose another 20 images and colorize them using the automatic methods and interactive methods mentioned above. We use original images as global inputs for  <cit.>, and use color themes extracted from original images as global inputs for our method. Local inputs are 3 to 20 colorful points randomly extracted from original images. Then we calculate the average PSNR of them as showed in Table <ref>. Through Table <ref>, we can see that results of automatic methods have similar performances, which have almost identical PSNRs. When we add global inputs, our method works much better and gets a higher PSNR.



For method with global inputs, the user guide method <cit.> shows good performance and get the highest PSNR. Our method also works well. But the PSNR of method is a bit lower than the user guide method <cit.>. This is possibly because we train our network to support four combination of inputs , the effect of local inputs is not optimized in our method.

For method with local inputs, the user guide method <cit.> shows good performance and get the highest PSNR.  Our method also works well. But the PSNR of method is a bit lower than the user guide method <cit.>. This is possibly also because we train our network to support four combination of inputs , the effect of local inputs is not optimized in our method. The optimization method  <cit.> gets a lower PSNR, probably because it was designed for strokes not point inputs.
  When both global inputs and local inputs  are used, our method get a higher PSNR, which is very close to the highest in Table <ref>.










 §.§ Colorization for past

We test our method on some historic images, which are white-and-black images. Although these images are different from our dataset, as that they may have rough edges and their picture quality may be unclear, they also can be colorized by our method. Compared with our method, other methods, which use local input only, maybe need a lot of local input points to get the same results. Some examples of coloring historic images are shown in Figure <ref>.




 §.§ Computation time

We test computation time of two image sizes on the NIVIDIA GTX1080Ti GPU. We take the average time of 100 computations on 50 images to get a reliable testing value, as shown in Table <ref>. The data shows that our method is comparable to real-time colorization.






 §.§ Limitations and discussion

Our model is trained by the Place dataset, which can not include everything in this world. If an gray-scale image contains certain contents, which is not learned from the data, the colors may be not assigned appropriately. In this case, the user may need to give more inputs to the system. For some images with unclear edges, the network may produce unexpected results, like color overflow on background. Users can add another local input points in the background region. In addition, When users add local input points with unusual colors, they may have to add more points to achieve the desired effect or directly change to use an appropriate color theme.






§ CONCLUSION AND FUTURE WORK

In this paper, we propose a novel interactive deep colorization method. By designing a suitable loss function, our method allows four combinations of input in a single network model, including no inputs, only global input, only local input, and both global input and local input. Furthermore, our global input is a color theme with variable number of colors, which is more easy and straightforward for user to use. Together with recommendation system, our method can reduce time and cost of coloring images for users. In addition, we evaluate our method on images of outdoor, human picture, past pictures, and show that it can produce satisfactory results for all them.
In the future, we would like to extend our method to gray-scale video sequence.













IEEEbib


10

Levin2004Colorization
Anat Levin, Dani Lischinski, and Yair Weiss,
“Colorization using optimization,”
in SIGGRAPH, 2004, pp. 689–694.

HORIUCHI-Example-2005
T. Horiuchi and H. Kotera,
“Colorization for monochrome image with texture,”
in Color Imaging Conference, 2005, pp. 245–250.

Huang2005An
Yi Chin Huang, Yi Shin Tung, Jun Cheng Chen, Sung Wen Wang, and Ja Ling Wu,
“An adaptive edge detection based colorization algorithm and its
  applications,”
in ACM MM, 2005, pp. 351–354.

yatziv2006fast
Liron Yatziv and Guillermo Sapiro,
“Fast image and video colorization using chrominance blending,”
Image Processing, IEEE Transactions on, vol. 15, no. 5, pp.
  1120–1129, 2006.

Luan2007Natural
Qing Luan, Fang Wen, Daniel Cohen-Or, Lin Liang, Ying Qing Xu, and Heung Yeung
  Shum,
“Natural image colorization,”
in Eurographics Conference on Rendering Techniques, 2007, pp.
  309–320.

Kawulok-2010
M. Kawulok and B. Smolka,
“Competitive image colorisation,”
in Proceedings of 17th International Conference on Image
  Processing, 2010, pp. 405–408.

Yao-Bayesian-2011
Chen Yao, Xiaokang Yang, Li Chen, and Yi Xu,
“Image colorization using bayesian nonlocal inference,”
Journal of Electronic Imaging, vol. 20, no. 2, pp.
  023008–1–023008–6, 2011.

Kawulok-Textural-2011
Michal Kawulok, Jolanta Kawulok, and Bogdan Smolka,
“Textural features for scribble-based image colorization,”
Computer Recognition Systems 4, vol. 95, pp. 269–278, 2011.

Welsh2002Transferring
Tomihisa Welsh, Michael Ashikhmin, and Klaus Mueller,
“Transferring color to greyscale images,”
in Conference on Computer Graphics and Interactive Techniques,
  2002, pp. 277–280.

Chang2005Example
Youngha Chang, Suguru Saito, Keiji Uchikawa, and Masayuki Nakajima,
“Example-based color stylization of images,”
Acm Transactions on Applied Perception, vol. 2, no. 3, pp.
  322–345, 2005.

Chia2011Semantic
Yong Sang Chia, Shaojie Zhuo, Raj Kumar Gupta, Yu Wing Tai, Siu Yeung Cho, Ping
  Tan, and Stephen Lin,
“Semantic colorization with internet images,”
ACM TOG, vol. 30, no. 6, pp. 1–8, 2011.

Gupta2012Image
Raj Kumar Gupta, Yong Sang Chia, Deepu Rajan, Ee Sin Ng, and Zhiyong Huang,
“Image colorization using similar images,”
in ACM MM, 2012, pp. 369–378.

liu-2008-intrinsic
Xiaopei Liu, Liang Wan, Yingge Qu, Tien-Tsin Wong, Stephen Lin, Chi-Sing Leung,
  and Pheng-Ann Heng,
“Intrinsic colorization,”
ACM Transactions on Graphics (SIGGRAPH Asia 2008 issue), vol.
  27, no. 5, pp. 152:1–152:9, December 2008.

Cheng2015Deep
Zezhou Cheng, Qingxiong Yang, and Bin Sheng,
“Deep colorization,”
in ICCV, 2015, pp. 415–423.

DBLP:journals/corr/NguyenMT16
Tung Duc Nguyen, Kazuki Mori, and Ruck Thawonmas,
“Image colorization using a deep convolutional neural network,”
CoRR, vol. abs/1604.07904, 2016.

Limmer2017Infrared
Matthias Limmer and Hendrik P. A. Lensch,
“Infrared colorization using deep convolutional neural networks,”
in ICMLA, 2017, pp. 61–68.

Efros2017Real
Richard Zhang, Jun-Yan Zhu, Phillip Isola, Xinyang Geng, Angela S Lin, Tianhe
  Yu, and Alexei A Efros,
“Real-time user-guided image colorization with learned deep
  priors,”
ACM TOG, vol. 9, no. 4, 2017.

Wang:2010:DIC:1882262.1866172
Baoyuan Wang, Yizhou Yu, Tien-Tsin Wong, Chun Chen, and Ying-Qing Xu,
“Data-driven image color theme enhancement,”
TOG, vol. 29, no. 6, pp. 146:1–146:10, December 2010.

Li2008ScribbleBoost
Y. Li, E. Adelson, and A. Agarwala,
“Scribbleboost: adding classification to edge-aware interpolation of
  local image and video adjustments,”
in Nineteenth Eurographics Conference on Rendering, 2008, pp.
  1255–1264.

Xu09sigasia
Kun Xu, Yong Li, Tao Ju, Shi-Min Hu, and Tian-Qiang Liu,
“Efficient affinity-based edit propagation using k-d tree,”
ACM TOG, vol. 28, no. 5, pp. 118:1–118:6, 2009.

Chen2012Manifold
Xiaowu Chen, Dongqing Zou, Qinping Zhao, and Ping Tan,
“Manifold preserving edit propagation,”
ACM TOG, vol. 31, no. 6, pp. 132, 2012.

Chang:2015:PPR
Huiwen Chang, Ohad Fried, Yiming Liu, Stephen DiVerdi, and Adam Finkelstein,
“Palette-based photo recoloring,”
ACM TOG, vol. 34, no. 4, jul 2015.

Endo2016DeepProp
Yuki Endo, Satoshi Iizuka, Yoshihiro Kanamori, and Jun Mitani,
“Deepprop: Extracting deep features from a single image for edit
  propagation,”
Computer Graphics Forum, vol. 35, no. 2, pp. 189–201, 2016.

sangkloy2016scribbler
Patsorn Sangkloy, Jingwan Lu, Chen Fang, FIsher Yu, and James Hays,
“Scribbler: Controlling deep image synthesis with sketch and
  color,”
CVPR, 2017.

DBLP:journals/corr/Frans17
Kevin Frans,
“Outline colorization through tandem adversarial networks,”
CoRR, vol. abs/1704.08834, 2017.

paintschainer.preferred.tech
Preferred Networks,
“Paints chainer,”
  <http://paintschainer.preferred.tech/webui/howto/>,
Accessed 2017.

Erik2001Color
Erik Reinhard, Michael Ashikhmin, Bruce Gooch, and Peter Shirley,
“Color transfer between images,”
IEEE Computer Graphics and Applications, vol. 21, no. 5, pp.
  34–41, 2001.

Li2007Learning
Cheng Li and S. V. N. Vishwanathan,
“Learning to compress images and videos,”
in ICML, 2007, pp. 161–168.

Charpiat2008Automatic
Guillaume Charpiat, Matthias Hofmann, and Bernhard Sch?lkopf,
“Automatic image colorization via multimodal predictions,”
in ECCV, 2008, pp. 126–139.

Morimoto2009Automatic
Yuji Morimoto, Yuichi Taguchi, and Takeshi Naemura,
“Automatic colorization of grayscale images using multiple images on
  the web,”
in SIGGRAPH, 2009, pp. 1–1.

Deshpande2015Learning
Aditya Deshpande, Jason Rock, and David Forsyth,
“Learning large-scale automatic image colorization,”
in ICCV, 2015, pp. 567–575.

IizukaSIGGRAPH2016
Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa,
“Let there be Color!: Joint End-to-end Learning of Global and Local
  Image Priors for Automatic Image Colorization with Simultaneous
  Classification,”
TOG, vol. 35, no. 4, 2016.

Zhang2016Colorful
Richard Zhang, Phillip Isola, and Alexei A. Efros,
“Colorful image colorization,”
in ECCV, 2016, pp. 649–666.

Varga2017Fully
Domonkos Varga and Tams Szirnyi,
“Fully automatic image colorization based on convolutional neural
  network,”
in ICPR, 2017, pp. 3691–3696.

Zhao2016Retracted
Yili Zhao, Dan Xu, and Yan Zhang,
“Retracted chapter: Image colorization using convolutional neural
  network,”
in IGTA, 2016, pp. 238–244.

larsson2016learning
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich,
“Learning representations for automatic colorization,”
in ECCV, 2016.

DBLP:journals/corr/IsolaZZE16
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros,
“Image-to-image translation with conditional adversarial networks,”
CoRR, vol. abs/1611.07004, 2016.

Liu2014AutoStyle
Yiming Liu, Cohen Michael, Uyttendaele Matt, and Rusinkiewicz Szymon,
“Autostyle: Automatic style transfer from image collections to
  users' images,”
in EGSR, 2014, pp. 21–31.

Simonyan2014Very
Karen Simonyan and Andrew Zisserman,
“Very deep convolutional networks for large-scale image
  recognition,”
Computer Science, 2014.

Yan2014Automatic
Zhicheng Yan, Hao Zhang, Sylvain Paris, Sylvain Paris, and Yizhou Yu,
“Automatic photo adjustment using deep neural networks,”
ACM TOG, vol. 35, no. 2, pp. 11, 2014.

Simo2016Learning
Edgar Simo-Serra, Satoshi Iizuka, Kazuma Sasaki, and Hiroshi Ishikawa,
“Learning to simplify: fully convolutional networks for rough sketch
  cleanup,”
ACM TOG, vol. 35, no. 4, pp. 121, 2016.

pathakCVPR16context
Deepak Pathak, Philipp Krähenbühl, Jeff Donahue, Trevor Darrell, and Alexei
  Efros,
“Context encoders: Feature learning by inpainting,”
2016.

Varga2017Automatic
Domonkos Varga,
“Automatic cartoon colorization based on convolutional neural
  network,”
in International Workshop on Content-Based Multimedia Indexing,
  2017, p. 28.

Ronneberger2015U
Olaf Ronneberger, Philipp Fischer, and Thomas Brox,
“U-net: Convolutional networks for biomedical image segmentation,”
in MICCAI, 2015, pp. 234–241.





