Generating High Quality Visible Images from SAR Images Using CNNs
Puyang Wang Vishal M. Patel













Generating High Quality Visible Images from SAR Images Using CNNs
    Puyang Wang  Vishal M. Patel 

Department of Electrical and Computer Engineering

Rutgers, The State University of New Jersey

94 Brett Rd, Piscataway, NJ 08854

{puyang.wang, vishal.m.patel}@rutgers.edu

    December 30, 2023
================================================================================================================================================================================================================





We propose a novel approach for generating high quality visible-like images from Synthetic Aperture Radar (SAR) images using Deep Convolutional Generative Adversarial Network (GAN) architectures.  The proposed approach is based on a cascaded network of  convolutional neural nets (CNNs) for despeckling and image colorization.  The cascaded structure results in faster convergence during training and produces high quality visible images from the corresponding SAR images.   Experimental results on both simulated and real SAR images show that the proposed method can produce visible-like images better compared to the recent state-of-the-art deep learning-based methods. 



Synthetic aperture radar image, despeckling, colorization. 





§ INTRODUCTION


Synthetic aperture radar (SAR) is a coherent radar imaging technology which is capable of producing high-resolution images of targets and landscapes.  Due to its ability to capture images both at night and in bad weather conditions, SAR imaging has several advantages compared to optical and infrared systems.
However, SAR images are often difficult to interpret mainly due to the following two reasons. 


  * They are contaminated by multiplicative noise known as speckle.   Speckle is caused by the constructive and destructive interference of the coherent returns scattered by small reflectors within each resolution cell <cit.>.   

  * Processed SAR images are often grayscale and they do not contain any color information.  

 These two issues often make the processing and interpretation of SAR images very difficult for both human interpreters and computer vision systems.     Hence, despeckling and proper colorization are important for semantically interpreting the reflectivity field in SAR imaging.  

Assuming that the SAR image is an average of L looks, the observed SAR image Y is related to the noise free image X by the following multiplicative model <cit.>

    Y = F⊙ X,

where F is the normalized fading speckle noise random variable and ⊙ denotes the element-wise multiplication.  One common assumption on F is that it follows a Gamma distribution with unit mean and variance 1/L and has the following probability density function <cit.>

    p(F) = 1/Γ(L)L^LF^L-1e^-LF,

where Γ(·) denotes the Gamma function and F ≥ 0, L ≥ 1.  





Based on the above SAR observation model, various methods have been developed in the literature to suppress speckle.  These include multi-look processing <cit.>, filtering methods <cit.>, wavelet-based despecking methods <cit.>, SAR block-matching 3D (SAR-BM3D) algorithm <cit.>, Total Variation (TV) methods <cit.>, and deep learning-based methods <cit.>.   
Note that some of these methods apply homomorphic processing in which the multiplicative noise is transformed into an additive noise by taking the logarithm of the observed data <cit.>.  

Although state-of-the-art SAR image despecking algorithms such as SAR-BM3D and wavelet-based methods are able to generate despeckled SAR images with sharp edges, the resulting despeckled images are often difficult to interpret due to their grayscale nature.  For example, even after despecking, it is difficult to distinguish between  sandy land and grass field due to the grayscale nature of SAR images.  Hence, generating a visible-like image from a SAR image is not only an interesting problem but also important for semantic segmentation and interpretation of SAR images. This problem shares some similarities with image colorization. However, there are some notable differences. First, in the image colorization domain (grayscale image to RGB) the luminance is directly given by grayscale input, so only the chrominance needs to be estimated. 
Secondly, in the case of colorization techniques, in general noiseless grayscale images are given as input to
obtain the RGB images. But in the case of SAR images, the input will have speckle and the expected output is the clean visible-like image with three RGB channels.

In this paper, we develop a deep learning-based method, called SAR-GAN, for the problem of SAR image to high quality visible image translation where we map a single channel noisy SAR image into a visible-like RGB image.   Figure <ref> shows a sample output from our SAR-GAN method.  Given a simulated speckled SAR image shown in Figure <ref> (a), SAR-GAN can generate not only the despeckled image but also the visible-like image as shown in  Figure <ref> (b).   As can be seen by comparing Figure <ref> (a) and Figure <ref> (b), that our method is able to simultaneously denoise and colorize the simulated SAR image reasonably well. 





§ PROPOSED METHOD

In this section, we provide details of the proposed SAR-GAN method in which we aim to learn a mapping from input speckled SAR images to visible images for both noise removal and colorization.  The proposed method consists of  three main components: despeckling sub-network G_D, colorization sub-network G_C and generative adversarial learning. The primary goal of the despeckling sub-network is to restore a clean image from a noisy observation. The colorization  sub-network then transforms the despeckled image into a visible image. Inspired by recent works on using generative adversarial learning for image colorization, we add the adversarial loss by introducing a discriminator network D.   The adversarial loss, empirically, can in principle become aware that gray looking outputs are unrealistic, and encourage a wider color distribution. 
The composition of the two sub-networks, despeckling and colorization, forms the generator G in a typical generative adversarial network (GAN) framework as follows:

    G = G_C ∘ G_D.


The overall structure of the proposed SAR-GAN method containing two sub-networks and the training procedure is shown in Figure <ref>, where black arrow lines indicate data flows and red arrow lines denote network parameter updating.  The detailed architectures of both sub-networks and loss functions are discussed in the following subsections.





 §.§ Despeckling Network

 The detailed architecture of the despeckling sub-network is shown in Figure <ref>, where Conv, BN and ReLu stand for Convolution and Batch Normalization and Rectified Linear Unit, respectively.  The despeckling CNN is adopted from our previous work <cit.> on SAR image restoration. Using a specific CNN architecture, we learn a mapping from an input SAR image into a despeckled image.  One possible solution to the despeckling problem would be to transform the image into a logarithm space and then learn the corresponding mapping via CNN <cit.>. However, this approach needs extra steps to transfer the image into a logarithm space and from a logarithm space back to an image space. As a result, the overall algorithm can not be learned in an end-to-end fashion. To address this issue, a division residual method is leveraged in our method where a noisy SAR image is viewed as a product of speckle with the underlying clean image (i.e. (<ref>)).  By incorporating the proposed component-wise division residual layer into the network,  the convolutional layers are forced to learn the speckle component during the training process.  In other words, the output before the division residual layer represents the estimated speckle.  Then, the despeckled image is obtained by simply dividing the input image by the estimated speckle.  

The noise-estimating part of despeckling sub-network consists of 8 convolutional layers (along with batch normalization and ReLU activation functions), with appropriate  zero-padding to make sure that the output of each layer shares the same dimension with that of the input image. Batch normalization is added to alleviate the internal covariate shift by incorporating a normalization step and a scale and shift step before the nonlinearity in each layer. Each convolutional layer (except for the last convolutional layer) consists of 64 filters with stride of one. Then, the division residual layer with skip connection divides the input image by the estimated speckle. A hyperbolic tangent layer is stacked at the end of the network which serves as a non-linear function. 







 §.§ Colorization Network

Deep learning-based image colorization has been studied over the last couple of years <cit.>, <cit.>. The key part of an image colorization neural network is to fully leverage the contextual information of an image for color translation. To extract and utilize the contextual information, one common way in deep learning is to use an encoder-decoder architecture in which an input image is encoded into a set of feature maps in the middle of the network. However, such a network requires that all information flow passes through all the layers.  For the image colorization problem, the sharing of low-level information between the input and output is important since the input and output should share the location of prominent edges. For the above reason, we add skip connections, following the general shape of an encoder-decoder CNN <cit.> as shown in Figure <ref>. 


The colorization sub-network forms a symmetric encoder-decoder with 8 convolution layers and 3 skip connections. For each convolution layer, the kernel size is 3 × 3. Note that the n64s1 in Figure <ref> and <ref> stands for 64 feature maps with one stride. 




 §.§ Loss Functions

In a SAR image translation problem, it is important that the output image is noise free and realistic. One common loss function used in many image translation problems is the L_1 loss. Given an image pair (X, Y), where Y is the noisy input image and X is the corresponding ground truth, the per-pixel L_1 loss is defined as

    L_L_1(Y,X;G) = 1/CWH∑_w=1^W∑_h=1^HG(Y^c,w,h)-X^c,w,h_1,

where G is the learned network and G(Y^c,w,h) is the filtered image. 
Note that we have assumed that X and Y are of the same size C× W × H where C stands for the number of color channels.
In this case, the network is trained to minimize the L_1 distance between the output and the ground truth on the training set. Although the L_1 loss has been shown to be very effective for image de-noising problem, it will incentivize an average, grayish color when it is uncertain which of several plausible color values a pixel should take on. In particular, L_1 will be minimized by choosing the median of the conditional probability density function over possible colors. Hence, the L_1 loss alone is not suitable for image colorization. Recent studies have shown adversarial loss, on the other hand, can in principle become aware that gray looking outputs are unrealistic, and encourage matching the true color distribution. 

Given a set of N despeckled and colorized images,  {X̂^c,w,h_i}_i=1^N, generated from the generator G, the adversarial loss to guide the generator is defined as 

    L_A(X̂; D, G) = -1/N∑_i=1^Nlog(D(X̂^c,w,h_i)),

where X̂^c,w,h_i=G(Y^c,w,h_i).  
One of the issues with the adversarial loss is that it does not rely on the ground truth X.   Hence, the results often contain artifacts that are not present in the clean ground truth image.  It tries to make the 'style' of the output closer to the training images.  


Considering the pros and cons of both losses, we combine the per-pixel L_1 loss and the adversarial loss together with appropriate weights to form our new refined loss function. The proposed loss function is defined as follows

    L_D = L_L1(gray(Y), gray(X); G_D),
       L_C = L_L1(Y, X ;G_C)+ λ_a L_A(X̂; D, G_C),
       L = L_D + L_C,

where gray(X) and gray(Y) are the corresponding grayscale versions of ground truth X and noisy Y with single channel, respectively.  Here, L_D and L_C are loss functions for despeckling and colorization sub-network, respectively. The overall function L is the sum of L_D and L_C. The L_1 loss in (<ref>) makes the despeckling network G_D learn a mapping between the speckled input and clean ground truth. Loss function L_C for the colorization sub-network is a weighted sum of  L_1 and adversarial loss. Note that, for SAR images, the number of color channels is equal to 1. Hence, the dimension of input Y and G_D(Y) should be 1× W × H and 3× W × H for X and G(Y). λ_a is a pre-defined weight for adversarial loss to balance the scale difference between losses. Because of the single combined loss function L we are able to train the network G which contains two sub-networks in an end-to-end fashion.








§ EXPERIMENTAL RESULTS

To evaluate the effectiveness and performance of our proposed method, we present and compare results of our SAR-GAN with others methods. Since no similar work on despeckling and colorization of SAR images simultaneously has been done, we compare the performance of our method with that of the
two CNN methods (CNN <cit.> and pix2pix <cit.>) and their combinations with the state-of-the-art despeckling algorithm SAR-BM3D <cit.>.  For all the compared methods, parameters are set as suggested in their corresponding papers.  
For the basic CNN method, we adopt the network structure proposed in <cit.> and train the  network using the same training dataset as used  to train our network.   

To train the proposed SAR-GAN network, we generate a dataset that contains 3292 image pairs. Training images are collected from the scraped Google Maps images <cit.> and the corresponding speckled images are generated using (<ref>).  All images are resized to 512 × 512. The entire network is trained  using the ADAM optimization method  <cit.>, with mini-batches of size 12 and learning rate of 0.0002. During training, we set λ_a = 0.1.The architecture of the discriminator D is adapted from that in <cit.>.




 




 §.§ Despeckling Performance

One key part of generating high quality visible images from SAR images is about removing as much speckle as possible meanwhile retaining the fine details. Therefore, we perform an experiment comparing despeckling performance of the proposed SAR-GAN and other SAR image despeckling algorithms including the state-of-the-art SAR-BM3D on synthetic SAR images. The outputs of the despeckling network G_D are used for comparison.

We randomly selected 85 speckled images out of all images in the dataset.  The remaining images are used for training the network.  Experiments are carried out on three different noise levels. In particular, the number of looks L is set equal to be 1, 4 and 10, respectively.  The Peak Signal to Noise Ratio (PSNR), Structural Similarity Index (SSIM) <cit.>, Universal Quality Index (UQI) <cit.>, and Despeckling Gain (DG) <cit.> are used to measure the denoising performance of different methods.  Average results of 85 test images corresponding this experiment are shown in Table <ref>.  As can be seen from this table, in all three noise levels, SAR-GAN provides the best performance compared to the other despeckling methods.







 §.§ Results on Synthetic Images

The despecking as well as colorization results on a synthetic image corresponding to different methods are shown in Figure <ref>.  The details of the four compared methods are as follows:

	
  * CNN Network is adopted from <cit.> and trained with only the L_1 loss. The input and output are speckled image and generated visible image, respectively.
	
  * CNN w/ SAR-BM3D The input images are first despeckled by SAR-BM3D and then fed into the network which is trained on image pairs of grayscale image and the corresponding color image.
	
  * pix2pix The L_1+cGAN network in <cit.> trained with the L_1 and the adversarial losses. The input and output are speckled image and generated visible image, respectively.
	
  * pix2pix w/ SAR-BM3D Similar to CNN w/ SAR-BM3D, the colorization network is replaced by the pix2pix network.




From Figure <ref>, we can clearly see that our proposed SAR-GAN performs the best overall. Compared with Figure <ref> (e) and (g), our result in Figure <ref> (d) suffers from less artifacts because of better despeckling performance of the despeckling network.   Furthermore, from (f) and (h) we see that  SAR-BM3D helps to suppress speckle but at the cost of losing some detail information. Note that (e) and (f) both have some gray color in the final output.  We believe that this is mainly due to the use of only the L_1 loss in their networks.




 §.§ Results on Real SAR Images

Finally, we evaluate the performance of the proposed SAR-GAN on a real SAR image. Results are shown in Figure <ref>. The real SAR image shown in Figure <ref> (a) is from the Vancouver scene of RADARSAT-1 operating on the C band <cit.>. Parameters of RADATSAT-1 for the Vancouver scene are as follows:

Sampling Rate F_r is 32.317 MHz, pulse duration T_r is 41.7 μs and radar frequency f_0 is 5.3 GHz.   
Figure <ref> (d) is the satellite image captured on the same date as in (a). By comparing Figure <ref> (c) and (d), we can clearly see that the proposed SAR-GAN is capable of generating high quality visible-like image from a real SAR image. 





§ CONCLUSION

We proposed a novel approach for generating high quality visible-like images from SAR images using GAN architectures. The proposed approach is based on the usage of a cascaded model for despeckling and colorization in a progressive way. The cascaded structure allows a fast convergence during the training and obtains a greater similarity between the given SAR image and the corresponding visible image. The proposed approach has been evaluated on both simulated and real SAR images and it is shown that the proposed  approach can provide better colorization compared to some of the recent deep learning-based methods.  


§ ACKNOWLEDGMENT

This work was supported by an ARO grant W911NF-16-1-0126.
IEEEtran



