Multi-scale HSV Color Feature Embedding for High-fidelity NIR-to-RGB Spectrum Translation
Huiyu Zhai Mo Chen Xingxing Yang Gusheng Kang




Dimensional Crossover of Microscopic Magnetic Metasurfaces for Magnetic Field Amplification
    A. V. Silhanek
    April 29, 2024
===========================================================================================

empty
empty






The NIR-to-RGB spectral domain translation is a formidable task due to the inherent spectral mapping ambiguities within NIR inputs and RGB outputs. Thus, existing methods fail to reconcile the tension between maintaining texture detail fidelity and achieving diverse color variations. In this paper, we propose a Multi-scale HSV Color Feature Embedding Network (MCFNet) that decomposes the mapping process into three sub-tasks, including NIR texture maintenance, coarse geometry reconstruction, and RGB color prediction. Thus, we propose three key modules for each corresponding sub-task: the Texture Preserving Block (TPB), the HSV Color Feature Embedding Module (HSV-CFEM), and the Geometry Reconstruction Module (GRM). These modules contribute to our MCFNet methodically tackling spectral translation through a series of escalating resolutions, progressively enriching images with color and texture fidelity in a scale-coherent fashion. The proposed MCFNet demonstrates substantial performance gains over the NIR image colorization task. Code is released at: MCFNethttps://github.com/AlexYangxx/MCFNet.









§ INTRODUCTION




Compared with traditional RGB imaging systems, the near-infrared (NIR) imaging system can capture unique spectral reflectance details and is widely used in nighttime video surveillance <cit.>, object detection, material analysis <cit.>, remote sensing systems <cit.>, and other fields. Although NIR imaging (780–2500nm) has unique application values, human visual systems, and computer vision systems have become accustomed to processing optical information in the visible light band (380–780nm) <cit.>. Therefore, NIR-to-RGB spectrum translation has become a valuable research topic.

Recent development in deep learning brings great advancement to image translation tasks, like grayscale image colorization <cit.>. However, the progress of NIR-to-RGB spectral domain translation <cit.> lags behind, which is intrinsically a much more challenging task compared with grayscale colorization since the domain gaps between non-overlapping spectral bands introduce mapping ambiguity, which requires simultaneous estimation of brightness and chromaticity values (while grayscale image colorization only requires estimation of the latter).

We find that the mapping between NIR and RGB domains is highly unpredictable, e.g., similar pixel values in the NIR domain can have very different RGB values and vice versa, as shown in Fig. <ref>. This causes conventional image-to-image translation paradigms directly map NIR images into RGB domain using given mapping ground-truth RGB images as supervision to be prone to produce monotonous, if not erroneous, predictions as the optimization process will push the prediction to approximate the statistical average <cit.>. Although some transfer learning-based methods <cit.> utilize grayscale images as semantic guidance that improves the mapping performance, they neglect that NIR images contain subtler texture information than their grayscale and RGB counterparts, which should be well maintained and transferred to the final results. 



A straightforward question arises: how to effectively solve the mapping ambiguity between the source NIR and target RGB domains, and balance the fidelity and diversity of texture details and color variations? We attempt to solve this issue by decomposing the challenging task into three sub-tasks, including NIR texture maintenance, coarse geometry reconstruction, and RGB color prediction. To this end, we propose a Multi-scale HSV Color Feature Embedding Network (MCFNet), with three delicately designed modules carried out for each sub-task. Specifically, the Geometry Reconstruction Module (GRM) learns contextual information from NIR inputs at a coarse level. The HSV Color Feature Embedding Module (HSV-CFEM) converts NIR inputs into HSV (Hue, Saturation, Value) color space, which can effectively describe and distinguish different colors in the image. Compared with RGB color space, it can more effectively distinguish different color regions to correct color feature differences between the NIR and RGB domains. Meanwhile, it also serves as color guidance for GRM. In addition, to improve texture fidelity, the Texture Preserving Block (TPB) extracts the near-infrared Laplacian <cit.> component and injects it into the colorized output, which can also encourage GRM and HSV-CFEM to pay more attention to spectral energy and color information, respectively. 
Contributions of this work are summarized as follows:


 
  * We propose a multi-scale HSV color Feature embedding learning paradigm (MCFNet), which breaks down the challenging NIR-to-RGB spectral domain translation task into three sub-tasks with different perceptive fields to progressively colorize NIR images.
 
  * A HSV color feature embedding module is tailored for multi-scale incorporation of color features to contextual geometry information, promoting consistent information propagation from coarse-to-fine network scales.
 
  * An effective texture-fused module is proposed that adaptively fuses the high-frequency texture information with color features and refines the fused feature maps to generate prediction with better texture fidelity.
 




§ RELATED WORK





 §.§ GAN-based Methods

In recent years, with the development of deep learning, research on near-infrared colorized methods based on convolutional neural networks has shown diversification, especially methods with DCGAN (Deep Convolutional Generative Adversarial Network) <cit.> as the core model. Suarez et al. <cit.> employed an end-to-end Generative Adversarial Network approach, which, however, showed effectiveness solely when trained on paired images. Following this, Mehri et al. <cit.> tackled the challenge of colorized NIR images under unsupervised learning conditions using Cycle (Consistent Generative Adversarial Network). Yang et al. <cit.> introduced a multiscale dense connection CycleGAN to learn color transformation from the NIR domain to the RGB domain. Subsequently, Yan et al. <cit.> proposed a colorization method that combines texture feature learning and achieves good texture results, but its generalization performance still needs to be greatly improved. These methods take colorless NIR images as input and directly produce colorized NIR images through the generator in the adversarial network, with provided RGB images as ground truth. However, due to the differences between the NIR-to-RGB spectral domain conversion and the conversion between grayscale and RGB images, the network may make erroneous predictions during the learning process. 



 §.§ Transfer Learning-based Methods

Due to the nonlinearity of inter-domain mapping, Wang et al. <cit.> adopted transfer learning methods to obtain color and semantic information. Yang et al. <cit.> proposed a cooperative learning paradigm that incorporates grayscale image colorization to colorize NIR images. However, these methods overlook the importance of the original semantics of near-infrared images. Sun et al. <cit.> proposed a method combining SPADE (spatially adaptive normalization) <cit.> generator and grayscale approximation self-reconstruction. The core strategy is to extract color and texture features from different network branches, but it fails to effectively utilize the guidance of deep color semantics. Nevertheless, We found that SPADE can effectively extract semantic information. The design of SPADE aims to solve the coupling problem between style and semantic information in the task of semantic map-to-image translation.

The spatially adaptive mechanism that dynamically adapts to the semantic information of the input image during the generation process enables the generated color NIR images to effectively preserve the NIR texture and the RGB color in the synthesized results.








§ PROPOSED METHOD



As shown in Fig. <ref>, the whole model is trained in a CycleGAN <cit.> style, where key components are the colorization network C_A and the generator G_B. Colorized NIR images are generated using the colorization network C_A, and GAN loss is imposed by using the discriminator D_A to distinguish between actual images and false-colored NIR images. Generator G_B is used to restore color images to single-channel NIR images, and the corresponding discriminator D_B is responsible for determining their authenticity. Based on the U-net architecture <cit.>, proposed as all generators in the model.

The colorization network is the core of our entire framework. Our goal is to solve the mapping ambiguity between the source NIR and target RGB mapping domains, and balance between the fidelity and diversity of the texture details and the color variation. 

The methodology we adopt is to decompose the challenging spectrum translation task into three sub-tasks (NIR texture maintenance, coarse geometry reconstruction, and RGB color prediction), iterate the spectrum translation task through multiple resolutions to progressively infuse and propagate color and texture information with diversity and scale consistency. As shown in Fig. <ref>, our MCFNet contains three modules that correspond to each sub-task. Specifically, the Geometry Reconstruction Module (GRM) learns the contextual information of NIR inputs at a coarse level. Besides, the HSV Color Feature Embedding Module (HSV-CFEM) converts NIR inputs into HSV color space to align the color feature discrepancy between the NIR domain and the RGB domain. Meanwhile, the HSV-CFEM also serves as a color feature guidance for GRM. In addition, to improve texture fidelity, the Texture Preserving Block (TPB) will extract near-infrared Laplacian components. Finally, the information obtained from all branches is fused using SPADE <cit.> to obtain the final result. Taking a given single-channel NIR image (x_nir∈R^H× W×1) as input, the colorization network generates our final target three-channel color NIR image (y_rgb∈R^H× W× 3). Below, we will provide a detailed introduction to each branch.







 §.§ Texture Preserving Block 
 
The proposed method directly extracts texture from the input NIR image x_nir. We apply the Laplacian operator to process x_nir. The Laplacian operator is a classic image edge detection operator <cit.> that can highlight high-frequency information in the image, including texture features and edge information. This approach effectively extracts valuable texture features from NIR images, resulting in generated images with strong texture perception and reducing the loss of texture information, which in turn encourages the GRM and HSV-CFEM to focus more on spectrum energy and color information respectively, instead of texture details.



 §.§ HSV Color Feature Embedding Module 
 
As shown in Fig. <ref>, the module of HSV-CFEM in the method adopts a DCGAN <cit.> network structure. We first expand the input NIR image x_nir to a three-channel image and then transform it into the HSV image format to serve as the input for the generator G_C (HSV-CFEM). Generator G_C (HSV-CFEM) learns the HSV image format x_hsv of the ground truth RGB image, aiming to generate an HSV image x̂_hsv that closely resembles the real one. To correct the color feature differences between the NIR and RGB domains, the y_hsv will extract its color feature information and integrate it into the y^'_rgb generated by G_A (GRM) to generate the final color NIR image y_rgb. It is worth noting that, in addition to obtaining y_hsv, this part will also generate multiple corresponding scale color feature maps F_color during the learning process. These maps contain rich chromatic information that can be used to guide colorization. Embedding color features at multiple scales enriches the color representation of generated images, contributing to their natural appearance and vividness. We will provide a detailed introduction to this part in the next section and demonstrate its importance in Section 4.





 §.§ Geometry Reconstruction Module
 
As shown in Fig. <ref>, on the decoder side of G_A (GRM), for the connections with y_2 and y_1 <cit.>, we use bilinear interpolation to upscale y_4 by 4 times and 8 times, respectively. y_3 is also upscaled by 4 times and concatenated with the upscaled y_4 before being connected to y_1. Therefore, during the learning process of G_A (GRM), y_4, y_3, and y_1 play crucial roles throughout the entire learning process. In order to maximize the value of color space feature maps obtained from G_C (HSV-CFEM) in colorization guidance, we choose to generate color feature maps F_color corresponding to the scales of y_4, y_3, and y_1 in the generator G_C (HSV-CFEM). All F_color will extract their important color information through SPADE <cit.>, and after upsampling the corresponding size near-infrared feature map, they will be fused with it to obtain a new feature map. After completing this series of operations, the output of G_A (GRM) will yield the color NIR image y^'_rgb, but this result is not the final target yet. y^'_rgb still needs to integrate rich texture feature information and color feature information. As mentioned earlier, the feature information obtained from all branches will be further fused to yield the ultimate target y_rgb. The entire network adaptively integrates high-frequency texture information with color features, and refines the fused feature map to achieve consistent propagation of information from coarse to fine.



 §.§ Objectives

During the training phase, the following four types of losses are employed:

1) GAN Loss:  Taking C_A: NIR to RGB as an example, the GAN loss can be expressed as

    ℒ_gan(C_A,D_A,A,B)=𝔼_B∼ p_data(B)[log D_A(B)]
    
    + 𝔼_A∼ p_data(A)[log(1-D_A(C_A(A)))]

where 𝔼_A∼ p_data(A) and 𝔼_B∼ p_data(B) are the distributions of the NIR images and the ground truth RGB images, respectively. The objective of C_A is to generate color NIR images, while D_B aims to distinguish between the generated pseudo-color images C_A(A) and the real RGB images B.

2) Pair Consistent Loss: For all images generated by C_A and G_B, pair consistent loss is applied with their corresponding ground truth images. It can be computed as:


    ℒ_pair(C_A,G_B,A,B)=𝔼_A∼ p_paired_data(A)[||C_A(A)-B||]
    
    +𝔼_B∼ p_paired_data(B)[||G_B(B)-A||]




3) Cycle Consistency Loss: As mentioned earlier, all reconstructed images are subjected to cycle consistency loss with the original input images. The cycle consistency loss can be expressed as:

    ℒ_cyc(C_A,G_B)=𝔼_A∼ p_data(A)[||G_B(C_A(A))-A||]
    +𝔼_B∼ p_data(B)|||C_A(G_B(B))-B||


4) Edge Loss: This can help the model better learn the texture and shape features of real images. Edges are important features in images, often used to capture texture and shape information. By computing the difference between the edges of generated images and real images, the visual quality and realism of generated images can be enhanced. It can be expressed as:

    ℒ_edge(C_A,G_B,A,B) 
       =𝔼_A∼ p_paired_data(A)[||E_A(C_A(A))-E_A(B)||]] 
       +𝔼_B∼ p_paired_data(B)[||E_B(G_B(B)-E_B(A))||]

where E_A and E_B respectively represent the extraction of edge features for color images and single-channel images.

Full Objective: The total loss can be expressed as follows:

    ℒ_total=ℒ_gan+λ_cycℒ_cyc+λ_pairℒ_pair+λ_edgeℒ_edge

where λ_cyc, λ_pair and λ_edge are the weights for their corresponding targets.




§ EXPERIMENTS AND ANALYSIS




 §.§ Implementation Details

We trained our network using 372 NIR images paired with their corresponding RGB images from the VCIP dataset <cit.>.

We further employed data augmentation techniques for training, including random resizing, cropping, contrast adjustment, and mirroring of images. All images have a size of 256 × 256, normalized to the range of (0,1). The training process consisted of two stages: In the first stage, from epoch 1 to 250, we trained the model using only paired data. In the second stage, from epoch 250 to 1000, we trained the model using both paired and unpaired data in an alternating manner, gradually reducing the learning rate.






 §.§ Comparison Experiments


To demonstrate the performance of our method, we quantitatively and qualitatively compare our MCFNet method with NIR-GNN <cit.>, MFF <cit.>, SST <cit.>, and ATCycleGAN <cit.>. 

Quantitative Evaluation: Based on 28 paired images from the test dataset, PSNR, SSIM, AE, and LPIPS metrics are adopted to assess the quality of the generated color NIR images and the performance of our proposed network.

According to the results in Table <ref>, our network outperforms other methods in terms of PSNR, SSIM, AE, and LPIPS. Our PSNR value is the highest, AE value and LPIPS value are the lowest, and SSIM value is comparable to MSS network. Specifically, our network shows an increase of 0.75 in the PSNR metric, a decrease of 0.54 in the AE metric, and a decrease of 0.087 in the LPIPS metric.

Qualitative Evaluation: We further provide the visual results in Fig. <ref>, our method demonstrates the best visual performance.

The images obtained by the ATCycleGAN <cit.> may exhibit poor overall chromaticity and saturation (such as the first row). As for the MFF <cit.>, while it excels in preserving the texture features of objects, its color results (the first and second rows) are not as impressive. Our method also exhibits outstanding performance in texture feature preservation. As shown in Fig. <ref>, when compared with other methods, we can observe that in the boundary regions between objects, such as the boundary between the sky and mountains (image in the fourth line), and the boundary between mountains and rivers (the fifth line), the texture features of the sky are usually smooth, while those of the mountains may be varied and rugged. Due to the mismatch between texture features on both sides of objects, the network may produce discontinuous or unnatural transitions between the sky and mountains, leading to image blurriness. Our approach, however, enables the network to effectively align texture and color information during the learning process, enhancing the network's adaptability to different texture features during colorization. This effectively reduces the occurrence of such situations.




 §.§ Ablation Experiments

Texture Feature Fusion: Based on the results in Table <ref>, experiments conducted without texture feature fusion showed varying degrees of decrease in PSNR and AE. As depicted in Fig. <ref>, images generated without guidance from texture information (the first row of shrubs) exhibited blurry features. In contrast, images generated with texture feature fusion displayed a clearer representation of textures and colors.

Multi-scale Color Fusion: Utilizing multiscale intermediate GAN features to guide colorization enables the full exploitation of color and detail information within GAN features. Compared to not employing this method, the multiscale color feature maps contain more semantic and color information, capturing more detailed features (such as houses on hillsides), thereby generating richer and more realistic color NIR images.





HSV-CFEM: The lack of color guidance during learning may lead to unreasonable color distribution, resulting in generated images lacking naturalness and accuracy. This could manifest as oversaturated, distorted, or unnatural colors in the generated images (e.g., the mountains in the third row), and may even generate completely erroneous color spots. Employing color space for colorization guidance provides global color characteristic information, thereby restoring more vibrant colors.



§ CONCLUSION

In this paper, we propose a multi-scale HSV color feature embedding learning paradigm (MCFNet). This paradigm decomposes the challenging task of NIR to RGB spectrum domain transformation into three sub-tasks with different perceptual fields: near-infrared texture maintenance, coarse geometry reconstruction, and RGB color prediction. Corresponding to three modules: Texture Preservation Block (TPB), HSV Color Feature Embedding Module (HSV-CFEM), and  Geometry Reconstruction Module (GRM). The proposed MCFNet can generate high-fidelity colorization results, not only presenting vivid color restoration but also preserving fine texture information of near-infrared input. Moreover, our method outperforms other methods in experiments.

-12cm   
                                  
                                  
                                  
                                  
                                  






IEEEbib





