C2MSNet: A Novel approach for single image haze removal
Akshay Dudhane Subrahmanyam Murala
cs.CV

















C2MSNet: A Novel approach for single image haze removal
    Akshay Dudhane1  Subrahmanyam Murala2

Computer Vision and Pattern Recognition Lab

Indian Institute of Technology Ropar

India

12017eez0001@iitrpr.ac.in

2subbumurala@iitrpr.ac.in

    Accepted January 20, 2018. Received January 3, 2018; in original form October 30, 2017
==========================================================================================================================================================================================

empty



   Degradation of image quality due to the presence of haze is a very common phenomenon. Existing DehazeNet <cit.>, MSCNN <cit.> tackled the drawbacks of hand crafted haze relevant features. However, these methods have the problem of color distortion in gloomy (poor illumination) environment. In this paper, a cardinal (red, green and blue) color fusion network for single image haze removal is proposed. In first stage, network fusses color information present in hazy images and generates multi-channel depth maps. The second stage estimates the scene transmission map from generated dark channels using multi channel multi scale convolutional neural network (McMs-CNN) to recover the original scene. To train the proposed network, we have used two standard datasets namely: ImageNet <cit.> and D-HAZY <cit.>. Performance evaluation of the proposed approach has been carried out using structural similarity index (SSIM), mean square error (MSE) and peak signal to noise ratio (PSNR). Performance analysis shows that the proposed approach outperforms the existing  state-of-the-art methods for single image dehazing.





§ INTRODUCTION


Low visibility in bad weather is a foremost problem in the computer vision. Appearance of the object in a human vision system requires the reflection of the atmospheric light from the object surface to the human eyes. However, presence of atmospheric particles (haze, fog etc.) in the transmission medium causes visual degradation in the quality of image. Most of the computerized systems assume that the input images are captured in clear atmosphere. Whereas, outdoor images undergo visibility degradation, especially in the morning (in presence of haze, fog etc.). So, to remove atmospheric effect computer vision algorithm related to outdoor imaging should be acquainted about the outdoor weather effect (haze). To achieve haze removal there is a dire need to develop an algorithm which could enhance the image quality in real time.



§ RELATED WORK

Various approaches had been proposed such as 3D geometrical model <cit.>, polarisation filters <cit.>, fusion of multiple images of same scenery <cit.> etc. for a single image haze removal. However, in real time applications, collecting multiple images of same scenery is not always feasible. Further, with a simple and convincing assumptions, dehazing models were proposed by <cit.>.
Tan <cit.> assumed contrast of the haze free image must be higher than the hazy scene. So, with this aforementioned assumption they removed haze by maximizing the local contrast of the hazy image. This method <cit.> fails and create blocking artifacts when there is a depth discontinuity in the hazy image.
Another effective local approach, dark channel prior (DChP) was proposed by He <cit.>. According to DChP, most of the local image patches in outdoor haze free colour images are comprised of some pixels (dark pixels) having very low intensity among at least one of their colour channels. According to Codruta <cit.> haze density can be predicted not only with the cardinal channel (RGB) but also in the HSV color space. They used the hue disparity between hazy image and its semi inverse image to detect the haze content. Huang <cit.> proposed visibility restoration (VR) algorithm to reduce color distortion and deal with complex structures which incorporates the edge information in the transmission map obtained using DChP. Alternatively, Zhu proposed a colour attenuation prior (CAP) <cit.> which estimates the depth map through HSV colour space.
Approaches discussed above witnessed the correlation of haze content with the color models and their advantages. Hence, learning the combination of all the assumption would anticipate the haze density more accurately. Because, single assumption does not guaranteed to be precise always, which may fail in extreme vague environment and complex structures. So, to develop a more robust system, <cit.> adopted convolution neural networks to learn the mapping between input hazy image and corresponding transmission map. However, <cit.> introduces the color distortion in gloomy (poor illumination) environment. Because, these networks loose the cardinal channel information in the first convolutional layer itself. Also, assumption made by the He <cit.> allows to split the cardinal channels and anticipate the haze spread. 
Inspired from <cit.> in this paper, we have proposed a new network to integrate the color information with respect to the gloominess in the environment and multi-scale filter bank to learn the pixel to pixel mapping between input hazy image and transmission map. We have trained this network on 3,50,000 variety of patches. Also, proposed network is tested on more than one thousand real as well as synthetic images. 



§ BACKGROUND

In this section we have highlighted the approach proposed by He <cit.>: 


 §.§ Optical Model

Optical model (OM) is based on the physical properties of the transmission of light through the air medium. Many computer vision algorithms make use of the optical model to describe the image formation. Especially, in haze removal many researchers used OM to reconstruct the original scene <cit.>. As given in <cit.>, Eq. 1 describes the optical model.

    I( x )=R( x )Tr( x )+A( 1-Tr( x ) )

where, I(x) is the pixel intensity of hazy image, R(x) is the scene radiance, Tr(x) is the transmission medium and A is the global atmospheric light.

Procedure to estimate the transmission map(Tr) and global airlight(A) using dark chennel prior <cit.> is given below.  


 §.§ Dark channel prior

Though there are many existing drawbacks, assumption made by the He <cit.> exhibits a strong solution to estimate the depth information and haze density. Keeping dark channel prior as a basis one can learn a strong haze features to restore the original scene. As discussed in section 2, with the key assumption He <cit.> estimated the scene depth using Eq. 2,

    D( x )=l∈ P( x )Γ ( c∈{ r,g,b }Γ I^c( l ) )

where, D(x) represents the dark channel, Γ(·) represents the minimum filter, P(x) belongs to the local patch cantered at location x having size of 15× 15, c is the colour channel of the hazy image, I is the hazy image.
It can be understood from the Eq. 2, if I is the haze free image then D(x) will be closer to zero. Basically, dark channel grasps the color information and anticipates the haze density. So, assumption made by the He allows us to split the cardinal channels and fuse them to predict/learn the haze spread/density. 


 §.§ Transmission map

Transmission map estimated using Eq. 3,

    Tr( x )=1-ηcΓ ( l∈ P( x )Γ ( I^c( P )/A^c) )

where, Tr(x) is a estimated transmission map, η is a haze constant is set to 0.95, Γ(·) represents a minimum filter, P(x) belongs to the local patch cantered at location x and patch size is taken as 15× 15, c is the colour channel of the hazy image and I is the hazy image and A is an estimated global air light.He <cit.> used dark channel to predict the airlight. Locations of 0.1% brightest pixel from the dark channel are accessed from hazy image and mean of these intensities taken as a global air light. Now, scene radiance (haze free image) R(x) can be easily recovered by using Eq. (1).



§ PROPOSED METHOD

Even though the DChP predicts the haze density, it fails in presence of dense haze and introduces the color distortion. Because, single depth map can not estimate the spread of the haze accurately. Optical model suggests that the precise estimation of depth/transmission map is the key to recover the scene radiance. Hence, to recover the scene radiance, we have designed a novel cardinal  color  fusion  multi-scale  CNN  (C2MSNet). Proposed network is divided into two stages, in first stage, we estimate the multiple depth channels (MDCh)[C2MSNet is designed to learn the different combinations of possible priors, it generates multiple depth maps. Here multiple depth map word indicates the haze relevant features.] using cardinal color fusion CNN (C^2-NN) and in second stage, multi-channel multi-scale CNN is used to learn the haze density/spread with respect to the transmission map.


 §.§ Cardinal color fusion CNN (C2-NN)

As discussed in section 2, haze is proportional to the intensity variation in various color channels. This observation lead us to fuse the color information and to design a novel network for estimation of depth maps. Proposed C^2-NN consists of convolution and pooling layers. Figure 1(a) shows the network architecture. Proposed C^2-NN divided into three sub-blocks namely: channel-wise -{convolution, concatenation  and  pooling}. Assumptions made by the <cit.> allow us to learn the haze spread by spilt and fuse operation on the color channels. In order to fuse the color information, initially we split the cardinal channels and applied parallel convolution filter banks of size 3× 3× 1× 32. Further, channel-wise concatenation (CwC) is performed on filter response using Eq. (4).



    CwC_i=⋒{Ψ _i^R, Ψ _i^G, Ψ _i^B}

where, ⋒{·} represents the concatenation operation, Ψ_i^R, Ψ_i^G and Ψ_i^B represent the convolution responses of cardinal channels respectively and i ∈ {1,2,..,32} are the number of convolution filters.
Finally, the channel-wise max pooling is performed on output of the concatenation layer. Output dimensions of the C^2-NN are m× n× 32. Whereas, m and n represent the image height and width respectively.



 §.§ Multi-channel multi-scale CNN (McMs-CNN)

He assumed 15×15 sized patch to obtain a dark channel prior. However, this fixed size patch makes DChP scale variant and causes halo effects at depth discontinuity. So, to make proposed network scale invariant, we designed multi-scale filter banks followed by spatial pooling and non-linear activation unit to retain the structural information. The output of C^2-NN is given to the McMs-CNN which consists of five layers as follows: {multi scale  convolution→ multi scale  concatination
→ pooling→ convolution→ BiRelu}.
To collect the multi-scale information, we have used three different filter banks, each having 16 filters: 3× 3× 16;  5× 5× 16  and   7× 7× 16. The multi scale convolution layer refines the MDCh and the multiscale-concatenation operation is performed to conjoin the haze features at coarse level which results in 48 feature maps. Further, the combination of spatial pooling and convolution layer have been used to extract the local haze relevant features. The spatial max pooling is performed within 7× 7 region having stride of one. The convolution (conv3) layer consists of 48 filters of size 5× 5 resulting 5× 5× 48 filter bank. Finally, the non-linear activation function (BiReLU) <cit.> has been used to anticipate the transmission channel.



 §.§ Training of C2MSNet

To construct the training set of hazy images, we have used D-HAZY <cit.> dataset which is generated from NYU depth indoor image collection <cit.>. D-HAZY <cit.> contains depth map for each indoor hazy image. We have generated 2,00,000 hazy patches (size 64× 64) along with respective depth maps from D-HAZY <cit.> dataset. Even though this database consists huge number of hazy images with depth maps, it does not consists outdoor natural scenes. So, to learn the haze spread in natural hazy environment we have extracted 1,50,000 patches (size 64× 64) from ImageNet <cit.> dataset. The hazy image model given by Eq. 1 is used to generate the synthetic hazy patches from ImageNet database. In total, we have generated 3,50,000 hazy patches with respective depth and transmission maps. Further, these hazy patches are randomly sampled and 1,50,000 patches were taken out to train the proposed C2MSNet and 2,00,000 patches for validation of the network.
The C2MSNet is initialized with random weight parameters and trained using stochastic gradient descent (SDG) back propagation algorithm with a batch size of 64 and learning rate of 0.002. The mean square error (MSE) is used as the loss function. The weight parameters of C2MSNet are updated in 18 epochs on PC with 4.20 GHz Intel Core i7 processor and NVIDIA GTX 1080 8GB GPU. Figure 2 shows the training process of C2MSNet with aforementioned settings. Also, we have calculated the time complexity of different methods for single image haze removal. We run the different algorithms on the same machine as described above. Table 1 shows the execution time taken by various methods for single image haze removal. From Table 1, it is clear that the proposed network is computationally efficient as compared to existing algorithms[We have used executable codes of different methods made available by the respective authors on their websites. The image size considered for the evaluation is 100×100×3.]. 





§ EXPERIMENTAL RESULTS AND DISCUSSION

In this section, the proposed de-hazing network is tested with both qualitative and quantitative analysis. The structural similarity index (SSIM), mean square error (MSE) and peak signal to noise ratio (PSNR) are considered for the quantitative analysis. We have categorized the experiments into three parts: indoor synthetic hazy images with depth maps, synthetic outdoor images and real hazy images without depth maps.




 §.§ Exp. 1: Indoor synthetic hazy images

As mentioned in section 4.3, D-HAZY is a benchmark dataset for single image haze removal. It contains 1,449 hazy images along with their depth maps respectively.


  §.§.§ Quantitative analysis

The performance evaluation of the proposed network has been addressed by testing 642 randomly sampled images from D-HAZY <cit.> dataset. Also, we have compared the performance of C2MSNet with existing state-of-the-art methods <cit.> as shown in Table 2. The considered evaluation measures are average values of SSIM, MSE and PSNR over 642 images. From Table 2, it is evident that the C2MSNet outprforms the other existing methods for single image haze removal on D-HAZY <cit.> dataset.











  §.§.§ Qualitative analysis

Figure 3 illustrates the synthetic hazy images and their corresponding scene radiance, recovered using proposed network and existing <cit.> methods. Figure 3(D-E) witnessed to the color distortion in recovered scene radiance with the DehazeNet <cit.> and MSCNN <cit.> approaches. Specfically, in Figure 3(c,D-E), as input image contains little gloomy illumination in marked region, <cit.> introduce color distortion in such a manner that edges in that region are not separable to

our eyes. Whereas, the proposed method gives better result for all cases (Figure 3 (a-d)) without color distortion and with more haze removal. The reason behind this is <cit.> fail to estimate the scene transmission map accurately in the poor illumination region. Whereas, the proposed C2MSNet is able to estimate the scene transmission map which is very close to the ground truth (see in Figure 4).



 §.§ Exp. 2: Outdoor synthetic hazy images

Performance of the proposed network on outdoor hazy images has been analysed using ImageNet <cit.> database. ImageNet <cit.> consists verity of classes, among which we considered only two: buildings and beach. Because, these two classes consists considerable variation in natural scenes. Using 3,000 natural scene images from these two classes, we have generated outdoor synthetic hazy image (OHI) database. In this section, we have evaluated the performance of proposed network on generated OHI database. 


  §.§.§ Quantitative analysis

For quantitative analysis, one should have certain basis (ground truth) for the corresponding input. In order to analyse the proposed network on OHI database, transmission maps and synthetic hazy images have been generated as discussed in section 4.3, with the help of Eq. 1. We have used SSIM, MSE and PSNR to compare the proposed  C2MSNet with existing state-of-the-art methods/approaches <cit.>. Table 3 shows the average SSIM, MSE and PSNR on 3,000 images randomly selected from OHI database. Table 3 witnessed to noticeable increment in the performance evaluation parameters for single image haze removal using proposed method. In natural scenes, constant color shade in sky region causes the color distortion in recovered scene. However, increment in the result witnessed to the usefulness/robustness of cardinal color fusion network to avoid the color distortion.
  






  §.§.§ Qualitative analysis

Almost all existing methods in recent publications, gives acceptable result on outdoor hazy images (frequently used). It is difficult to sort them in descending order with respect to removal of haze and visual quality. So, to compare with existing dehazing methods, we have employed new set of synthetic hazy images from OHI database. These images comprised of constant shade of cardinal colors. So that, color distortion using existing methods can be easily observed. Figure 5 shows the visual comparison for outdoor image haze removal. As shown in Figure 5 (C,c-f), we can observe that the color distortion in the recovered scene by existing state-of-the-art methods <cit.>. Whereas, Figure 5 (A-F, g) (proposed method) shows the restored original scene without any color distortion.   


 §.§ Exp. 3: Real world hazy images



  §.§.§ Qualitative analysis

In this section, we have carried out only qualitative analysis. Because, it is very difficult to capture the photo of the same scene in hazy and haze free environment. So, in absence of haze free scene (ground truth) we could not conduct the quantitative analysis. For visual (qualitative) analysis, we have considered set of five frequently used (in recent publications) real hazy images. Figure 6 shows the visual comparison for haze removal from real hazy images. From Figure 6, it is observed that the DChP and CAP have more color distortion than the CNN approach. Reason behind this could be the failure of hazy model assumption. The proposed network overcomes this drawback by learning the haze spread.




§ CONCLUSION

In this paper, we proposed a novel C2MSNet for single image haze removal which overcomes the color distortion problem. Proposed network fuses the cardinal color channel and later estimates the transmission map using proposed multi-channel multi-scale CNN. To prove the robustness of proposed network, three databases namely: D-HAZY, OHI and set of real hazy images have been used. Quantitative analysis has been carried out using SSIM, MSE and PSNR. The proposed method also compared with the existing state-of-the-art methods for single image haze removal using visual analysis. Extensive analysis with the help of three experiments shows that the proposed network outperforms the other existing methods for single image haze removal.



§ ACKNOWLEDGEMENT

We would like to thank Mr. Sachin Chaudhary and Mr. Prashant Patil, research scholar from Computer Vision and Pattern Recognition (CVPR) Lab, IIT Ropar and Mr. Gajanan Galshetwar for their valuable suggestions in this work. We also thank to anonymous reviewers for their insightful comments and helpful suggestions to improve the quality of this paper. 







AAAA

ancuti2016d Ancuti, Cosmin and Ancuti, Codruta O and De Vleeschouwer, Christophe, D-HAZY: A dataset to evaluate quantitatively dehazing algorithms, Image Processing (ICIP), 2016 IEEE International Conference on,  pages 2226–2230, IEEE, 2016.
 
ancuti2010fast Ancuti, Codruta O and Ancuti, Cosmin and Hermans, Chris and Bekaert, Philippe, A fast semi-inverse approach to detect and remove the haze from a single image,
Asian Conference on Computer Vision,  pages 501–514,
Springer, 2010.
 
cai2016dehazenet Cai, Bolun and Xu, Xiangmin and Jia, Kui and Qing, Chunmei and Tao, Dacheng, Dehazenet: An end-to-end system for single image haze removal,
 IEEE Transactions on Image Processing, 25(11),  pages 5187–5198,
 IEEE, 2016.
 
609419 F. Cozman and E. Krotkov, Depth from scattering,
 Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition,  pages 801-806, June 1997.
 
deng2009imagenet Deng, Jia and Dong, Wei and Socher, Richard and Li, Li-Jia and Li, Kai and Fei-Fei, L, Imagenet: A large-scale hierarchical image database, Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on,  pages 248–255, IEEE, 2009.
     
fattal2008single Fattal, Raanan, Single image dehazing, ACM transactions on graphics (TOG), 27(3): 72, 2008.
          
he2011single He, Kaiming and Sun, Jian and Tang, Xiaoou, Single image haze removal using dark channel prior, IEEE transactions on pattern analysis and machine intelligence, 33(12),  pages 2341–2353, IEEE, 2011.

huang2014visibility Huang, Shih-Chia and Chen, Bo-Hao and Wang, Wei-Jheng, Visibility restoration of single hazy images captured in real-world weather conditions, IEEE Transactions on Circuits and Systems for Video Technology, 24(10),  pages 1814–1824, IEEE, 2014.
  
narasimhan2002vision Narasimhan, Srinivasa G and Nayar, Shree K, Vision and the atmosphere, International Journal of Computer Vision, 48(3),  pages 233–254, Springer, 2002.
  
nayar1999vision Nayar, Shree K and Narasimhan, Srinivasa G, Vision in bad weather, Computer Vision, 1999. The Proceedings of the Seventh IEEE International Conference on, 2,  pages 820–827, IEEE, 1999.
  
ren2016single Ren, Wenqi and Liu, Si and Zhang, Hua and Pan, Jinshan and Cao, Xiaochun and Yang, Ming-Hsuan, Single image dehazing via multi-scale convolutional neural networks, European Conference on Computer Vision, pages 154–169, Springer, 2016.
    
schechner2001instant Schechner, Yoav Y and Narasimhan, Srinivasa G and Nayar, Shree K, Instant dehazing of images using polarization, Computer Vision and Pattern Recognition, 2001. CVPR 2001. Proceedings of the   2001 IEEE Computer Society Conference on, I, pages I325–I322, IEEE, 2001.
  
1640996 S. Shwartz and E. Namer and Y. Y. Schechner, Blind Haze Separation, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06), 2, pages 1984-1991, IEEE, 2006.
    
silberman2012indoor Silberman, Nathan and Hoiem, Derek and Kohli, Pushmeet and Fergus, Rob, Indoor segmentation and support inference from rgbd images, Computer Vision–ECCV 2012, pages 746–760, Springer, 2012.

tan2008visibility Tan, Robby T, Visibility in bad weather from a single image, Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on, pages 1–8, IEEE, 2008.
    
    
zhu2014single Zhu, Qingsong and Mai, Jiaming and Shao, Ling, Single Image Dehazing Using Color Attenuation Prior, BMVC, 2014.
    
zhu2015fast Zhu, Qingsong and Mai, Jiaming and Shao, Ling, A fast single image haze removal algorithm using color attenuation prior, IEEE Transactions on Image Processing, 24(11) pages 3522–3533, IEEE, 2015.

ieee



