Adaptive Image Sampling using Deep Learning and its Application on X-Ray Fluorescence Image Reconstruction
Qiqin Dai Henry Chopp Emeline Pouyet Oliver Cossairt Marc Walton Aggelos K. Katsaggelos










































Journal of  Class Files, Vol. 11, No. 4, December 2012
Shell et al.: Bare Demo of IEEEtran.cls for Journals


























Adaptive Image Sampling using Deep Learning and its Application on X-Ray Fluorescence Image Reconstruction
    Qiqin Dai,	
		Henry Chopp,
        Emeline Pouyet,
        Oliver Cossairt,
        Marc Walton,
        and Aggelos K. Katsaggelos, Fellow, IEEE
Q. Dai, H. Chopp, O. Cossairt and A. K. Katsaggelos are with the Department
of Electrical Engineering and Computer Science, Northwestern University, Evanston, IL, 60208 USA. E. Pouyet and M. Walton are with the Northwestern University / Art Institute of Chicago Center for Scientific Studies in the Arts (NU-ACCESS), Evanston, IL, 60208 USA.

Manuscript received September 15, 2018.
    December 30, 2023
====================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================






This paper presents an adaptive image sampling algorithm based on Deep Learning (DL). It consists of an adaptive sampling mask generation network which is jointly trained with an image inpainting network. The sampling rate is controlled by the mask generation network, and a binarization strategy is investigated to make the sampling mask binary. In addition to the image sampling and reconstruction process, we show how it can be extended and used to speed up raster scanning such as the X-Ray fluorescence (XRF) image scanning process. Recently XRF laboratory-based systems have evolved into lightweight and portable instruments thanks to technological advancements in both X-Ray generation and detection. However, the scanning time of an XRF image is usually long due to the long exposure requirements (e.g., 100 μ s-1ms per point). We propose an XRF image inpainting approach to address the long scanning times, thus speeding up the scanning process, while being able to reconstruct a high quality XRF image. The proposed adaptive image sampling algorithm is applied to the RGB image of the scanning target to generate the sampling mask. The XRF scanner is then driven according to the sampling mask to scan a subset of the total image pixels. Finally, we inpaint the scanned XRF image by fusing the RGB image to reconstruct the full scan XRF image. The experiments show that the proposed adaptive sampling algorithm is able to effectively sample the image and achieve a better reconstruction accuracy than that of existing methods.





Adaptive sampling, convolutional neural network, X-Ray fluorescence, inpainting





















§ INTRODUCTION


















With the increasing demand for multimedia content, there has been more and more interest in visual data acquisition. Many visual data, such as Lidar depth map, scanning probe microscopy (SPM) image, XRF image, etc, is acquired by the time consuming raster scan process. Thus sampling techniques need to be investigated to speed up the acquisition process. Compressed sensing (CS) has shown that it is possible to acquire and reconstruct natural images under the Nyquist sampling rates <cit.>. Rather than full image acquisition followed by compression, CS combines sensing and compression into one step, and has the advantages of faster acquisition time, smaller power consumption, and lighter data throughput. Adaptive image sampling is a sub-problem of CS that aims for a sparse representation of signals in the image domain. In this paper, we present a novel adaptive image sampling algorithm based on Deep Learning and show its application to RGB image sampling and recovery. We also applied the proposed adaptive sampling technique to  speed up the raster scan process of XRF imaging, based on the correlation between RGB and XRF signals.

Irregular sampling techniques have long been studied in the image processing and computer graphics fields to achieve compact representation of images. Such irregular sampling techniques, such as stochastic sampling <cit.>, may have better anti-aliasing performance compared to uniform sampling intervals if frequencies greater than the Nyquist limit are present. Further performance improvement can be obtained if the sampling distribution is not only irregular but also adaptive to the signal itself. The limited samples should be concentrated in parts of the image rich in detail, so as to simulate human vision <cit.>. Several works have been reported in the literature on adaptive sampling techniques. An early significant work in this direction is made by Eldar et al. <cit.>. A farthest point strategy is proposed which permits progressive and adaptive sampling of an image. Later, Rajesh et al. <cit.> proposed a progressive image sampling technique inspired by the lifting scheme of wavelet generation. A similar method is developed by Demaret et al. <cit.> by utilizing an adaptive thinning algorithm. Ramponi et al. <cit.> developed an irregular sampling method based on a measure of the local sample skewness. Lin et al. <cit.> viewed grey scale images as manifolds with density and sampled them according to the generalized Ricci curvature. Liu et al. <cit.> proposed an adaptive progressive image acquisition algorithm based on kernel construction. Recently, Taimori et al. <cit.> investigated adaptive image sampling approaches based on the space-frequency-gradient information content of image patches.

Most of these irregular sampling and adaptive sampling techniques <cit.> need their own specific reconstruction algorithm to reconstruct the fully sampled signal. Furthermore, all these sampling techniques are model-based approaches relying on predefined priors, and according to our knowledge, no work has been done on utilizing machine learning techniques to design the adaptive sampling mask.

Inspired by the recent successes of convolutional neural networks (CNNs) <cit.> in high level computer vision tasks, deep neural networks (DNNs) emerged in addressing low level computer vision tasks as well <cit.>. For the task of image inpainting, Pathak et al. <cit.> presented an auto-encoder to perform context-based image inpainting. The inpainting performance is improved by introducing perceptual loss <cit.> and on-demand learning <cit.>. Iliadis et al. <cit.> utilized a deep-fully-connected network for video compressive sensing while also learning an optimal binary sampling mask <cit.>. However, the learned optimal binary sampling mask is not adaptive to the input video signals. According to our knowledge, no work has been made on generating the adaptive binary sampling mask for the image inpainting problem using deep learning.

With the proposed adaptive sampling algorithm, we can efficiently sample points based on the local structure of the input images. Besides the application of Progressive Image Transmission (PIT) <cit.>, compressed image sampling, image coding <cit.>, etc., we also show that the proposed adaptive sampling algorithm can speed up many raster scan processes such as XRF imaging. In detail, the binary sampling mask can be obtained using one modality (RGB image for example) of the target, and then applied on the raster scan process for another modality (XRF image for example). A detailed introduction to XRF imaging is in Section <ref>.

The contribution of this paper lies in the following aspects:


  
  * We proposed an effective way to binarize and control the sparseness of the output of a CNN.
  
  * We proposed an efficient network structure to generate the binary sampling mask and showed its advantages over other state-of-the-art adaptive sampling algorithms.
  
  * We proposed an adaptive sampling framework that can be applied to speed up many raster scan processes.
  
  * We proposed a fusion-based image reconstruction algorithm to restore the fully sampled XRF image.
  
  * We experimentally showed that the benefits of the adaptive sampling and the fusion-based inpainting algorithm are additive.


This paper is organized as follows: Section <ref> introduces XRF imaging with adaptive sampling. We illustrate the adaptive sampling mask design in Section <ref>. We describe the XRF image inpainting problem in Section <ref>. In Section <ref>, we provide the experimental results with both synthetic data and real data to evaluate the effectiveness of the proposed approach. The paper is concluded in Section <ref>.














§ XRF IMAGING USING ADAPTIVE SAMPLING



During the past few years, XRF laboratory-based systems have evolved to lightweight and portable instruments thanks to technological advancements in both X-Ray generation and detection. Spatially resolved elemental information can be provided by scanning the surface of the sample with a focused or collimated X-ray beam of (sub) millimeter dimensions and analyzing the emitted fluorescence radiation in a nondestructive in-situ fashion entitled Macro X-Ray Fluorescence (MA-XRF). The new generations of XRF spectrometers are used in the Cultural Heritage field to study the manufacture, provenance, authenticity, etc. of works of art. Because of their fast noninvasive set up, we are able to study large, fragile, and location-inaccessible art objects and archaeological collections. In particular, XRF has been used extensively to investigate historical paintings by capturing the elemental distribution images of their complex layered structure. This method reveals the painting history from the artist creation to restoration processes <cit.>.

As with other imaging techniques, high spatial resolution and high quality spectra are desirable for XRF scanning systems; however, the acquisition time is usually limited, resulting in a compromise between dwell time, spatial resolution, and desired image quality. In the case of scanning large scale mappings, a choice may be made to reduce the dwell time and increase the step size, resulting in noisy XRF spectra and low spatial resolution XRF images.

An example of an XRF scan is shown in Figure <ref> (a). Channel #582 - 602 corresponding to the Pb Lη XRF emission line was extracted from a scan of Jan Davidsz. de Heem's “Bloemen en insecten” painted in 1645 (housed at Koninklijk Museum voor Schone Kunsten (KMKSA) Antwerp). The image is color coded for better visibility. This XRF image was collected by a home-built XRF spectrometer (courtesy of Prof. Koen Janssens) with 2048 channels in spectrum and spatial resolution 680 × 580 pixels. This scan has a relatively short dwell time, resulting in low Signal-to-Noise Ratio (SNR), yet it still took 18 hours to acquire it. Faster scanning speed will be desirable for promoting the popularity of the XRF scanning technique, since the slow acquisition process impedes the use of XRF scanning instruments as high resolution widefield imaging devices. The RGB image of the painting of resolution 680 × 580 pixels is shown in Figure <ref> (b).







Image inpainting <cit.> is the process of recovering missing pixels in images. The XRF images are acquired through a raster scan process. We could therefore speed up the scanning process by skipping pixels and then utilizing an image inpainting technique to reconstruct the missing pixels. If we are to skip 80% of the pixels during acquisition (a 5x speedup), we could use a random sampling mask (shown in Figure <ref> (a)), or we could design one utilizing the available RGB image (shown in Figure <ref> (b)). The idea of the adaptive binary sampling mask is based on the assumption that the XRF image is highly correlated with the RGB image. We would like to allocate more pixels to the informative parts of the image, such as high frequency textures, sharp edges, and high contrast details, and spend fewer pixels on the uninformative parts of the image.

With the proposed adaptive sampling algorithm, we propose an image inpainting approach to speed up the acquisition process of the XRF image with the aid of a conventional RGB image, as shown in Figure <ref>. The proposed XRF image inpainting algorithm can also be applied to spectral images obtained by any other raster scanning processes, such as Scanning Electron Microscope (SEM), Energy Dispersive Spectroscopy (EDS), and Wavelength Dispersive Spectroscopy (WDS). First, the RGB image of the scanning target is utilized to generate the adaptive sampling mask. Then, the XRF scanner will scan the corresponding pixels according to the binary sampling mask. The speedup in acquisition is achieved since many pixels will be skipped. Finally, the subsampled XRF image is fused with the conventional RGB image to reconstruct the full scan XRF image, utilizing an image inpainting algorithm. For the fusion-based XRF image inpainting algorithm, similarly to our previous super-resolution (SR) approach <cit.>, we model the spectrum of each pixel using a linear mixing model <cit.>. Because the hidden part of the painting is not visible in the conventional RGB image, but it can be captured in the XRF image <cit.>, there is no direct one-to-one mapping between the visible RGB spectrum and the XRF spectrum. We model the XRF signal as a combination of the visible signal (on the surface) and the non-visible signal (hidden under the surface), as shown in Figure <ref>. We emphasize that while our framework is general enough to handle separation of visible and hidden layers, it easily handles the case of fully visible layers. To inpaint the visible component XRF signal, we follow an approach similar to the one applied to hyper-spectral image SR <cit.>. A coupled XRF-RGB dictionary pair is learned to explore the correlation between XRF and RGB signals. For the non-visible part, we inpaint its missing pixels using a standard total variation regularizer. Finally, the reconstructed visible and non-visible XRF signals are combined to obtain the final XRF reconstruction result. The input subsampled XRF image is not explicitly separated into visible and non-visible parts in advance. Instead, the whole inpainting problem is formulated as an optimization problem. By alternately optimizing over the coupled XRF-RGB dictionary and the visible/non-visible fully sampled coefficient maps, the fidelity of the estimated fully sampled output to both the subsampled XRF and RGB input signals is improved, thus resulting in a better inpainting output. Real experiments show the effectiveness of our proposed method in terms of both reconstruction error and visual quality of the inpainting result.


While there is a large body of work on inpainting conventional RGB images <cit.>, very little work has appeared in the literature on inpainting XRF images <cit.>, and there is no work on fusing a conventional RGB image during the inpainting process. Some learning based approaches <cit.> have been proposed to perform multi-modality image inpainting. However, due to the limited amount of high quality XRF image training data, they can not be applied yet. XRF image inpainting poses a particular challenge because the acquired spectrum signal usually has low SNR. In addition, the correlation among spectral channels needs to be preserved for the inpainted pixels. In our previous work on spatial-spectral representation for XRF image super-resolution <cit.>, the spatial resolution of the visible component XRF signal is increased by fusing an HR conventional RGB image while the spatial resolution of the non-visible part is increased by using a standard total variation regularizer <cit.>. Here we propose an XRF image inpainting algorithm by fusing an HR conventional RGB image, which can be regarded as an extension of our previous XRF SR approach.




§ ADAPTIVE SAMPLING MASK GENERATION UTILIZING CONVOLUTIONAL NEURAL NETWORK



In this section, we present our proposed adaptive sampling mask generation using a CNN. In other words, we describe the details of the “Sampling Mask Generation” block in Figure <ref>. We first formulate the problem of adaptive sampling mask design, followed by the presentation of the overall network architecture consisting of both the inpainting network and the mask generation network.



 §.§ Problem Formulation







As shown in Figure <ref>, we denote by z an input original image. Our mask generation network NetM produces a binary sampling mask m = NetM(z, c),  where c ∈ [0 1] is the predefined sampling percentage. The entries of m are equal to 1 for the sampled and 0 otherwise. The corrupted image z^' is obtained by




    z^' = z ⊙ m = z ⊙ NetM(z, c),
 where ⊙ is the element-wise product operation. The reconstructed image z̅ is obtained by the inpainting network NetE,


    z̅ = NetE(z^') = NetE(z ⊙ NetM(z, c)).


The overall pipeline is shown in Figure <ref>. We could regard the whole pipeline (Equation <ref>) as one network with input z and output z̅ and perform an end to end training. If we simultaneously optimize the mask generation network NetM and the inpainting network NetE according to the following loss function,


    (z) =  z - z̅_2 =  z - NetE(z ⊙ NetM(z, c)) _2,

NetM will perform an optimized adaptive sampling strategy according to the input image, and NetE will perform optimized image inpainting. After the mask has been generated by the network NetM, we can replace the inpainting network NetE with other image inpainting algorithms. The detailed network architecture of NetE and NetM are discussed in the following two subsections <ref> and <ref>, respectively. The detailed training procedure of NetM and NetE will be discussed in Section <ref>




 §.§ Deep Learning Network Architecture for Inpainting Network




The network architecture in <cit.> is used for the inpainting network, as shown in Figure <ref>. The network is an encoder-decoder pipeline. The encoder takes a corrupted image z^' of size 64 × 64 as input and encodes it in the latent feature space. The decoder takes the feature representation and outputs the restored image z̅=NetE(z^'). The encoder and decoder are connected through a channel-wise fully-connected layer. For the encoder, four convolutional layers are utilized. A batch normalization layer <cit.> is placed after each convolutional layer to accelerate the training speed and stabilize the learning process. The Leaky Rectified Linear Unit (LeakyReLU) activation <cit.> is used in all layers in the encoder.

The convolutional layers in the encoder only connect all the feature maps together, but there are no direct connections among different locations within each specific feature map. Fully-connected layers are then applied to handle this information propagation. To reduce the number of parameters in the fully connected layers, a channel-wise fully-connected layer is used to connect the encoder and decoder, as in <cit.>. The channel-wise fully connected layer is designed to only propagate information within activations of each feature map. This significantly reduces the number of parameters in the network and accelerates the training process.

The decoder consists of four deconvolutional layers <cit.>, each of which is followed by a ReLU activation except for the output layer. The tanh function is used in the output layer to restrict the pixel range of the output image. The series of up-convolutions and nonlinearities perform a nonlinear weighted upsampling of the feature produced by the encoder and generates an inpainted image of the target size (64 × 64).





 §.§ Deep Learning Network Architecture for the Mask Generation Network





According to our knowledge, no prior work has been reported on generating the adaptive binary sampling mask utilizing CNNs. The desired mask generation network NetM should satisfy the following criteria:


  
  * The output image should have the same spatial resolution as the input image.
  
  * The network architecture should be fully convolutional to handle arbitrary input sizes.
  
  * The output image should be binary.
  
  * The output image should have a certain percentage c of 1's.


Inspired by <cit.>, a network architecture with residual blocks is applied here, as shown in Figure <ref>. The network NetM consists of B residual blocks with identical layout. Following the method used in <cit.>, we use two convolutional layers with small 3×3 kernels and 64 feature maps followed by batch-normalization layers <cit.> and ParametricReLU <cit.> as the activation function. The network NetM is fully convolutional to handle an arbitrary input size. To keep the spatial dimensions at each layer the same, the images are padded with zeros. At the end of each residual block is a final elementwise sum layer, followed by a Sigmoid activation layer. Let us denote by L_ij the (i,j)^th element of L, which is the output of the Sigmoid activation layer, and mapped to the range [0 1]. The Mean Adjustment layer F is defined as



    D_ij = F(L_ij) = cL̅× L_ij,

where D_ij is the (i,j)^th element of matrix D ∈𝐑^2, which is the output of the Mean Adjustment layer, and L̅ is the mean of L. Then D_ij∈ [0 1] and the mean value of D, denoted by D, will be equal to c. Finally, the Bernoulli distribution Ber() is applied to binarize the values of D; that is,


    B_ij = Ber(D_ij) =
    
    1,     p = D_ij
    
    0,     p = 1 - D_ij
    .

Notice that

    B≈1/N^2∑_i=1^N∑_j=1^N E(B_ij) = 1/N^2∑_i=1^N∑_j=1^N D_ij = c,

where N^2 is the total number of pixels of L and E(B_ij) is the expected value of B_ij. Therefore, B is binary matrix with mean value equal to c, implying that it has c percent of 1's.

Since applying the function Ber(F(·)) on the input L will make the output of the network be binary and have c percent of 1's, we then make it the last Binarization layer activation function. Notice that function Ber(D) is not continuous and its derivatives do not exist, making the back propagation optimization during training impractical. We use its expected value D to approximate it during training and apply the original function Ber(D) during testing.





§ SPATIAL-SPECTRAL REPRESENTATION FOR X-RAY FLUORESCENCE IMAGE INPAINTING



In this section, we propose the XRF image inpainting algorithm by fusing it with a conventional RGB image, detailing the “Proposed Inpainting Algorithm” block in Figure <ref>. The proposed fusion style inpainting approach has similarities with our previous fusion style SR approach <cit.>. We first formulate the XRF image inpainting problem, then demonstrate our proposed solution to this inpainting problem.



 §.§ Problem Formulation



As shown in Figure <ref>, we are seeking the estimation of a reconstructed XRF image Y̅∈ℝ^W × H × B that is fully sampled, with W, H, and B the image width, height, and number of spectral bands, respectively. We have two inputs: a subsampled XRF image X̅∈ℝ^W × H × B with the known binary sampling mask S̅∈ℝ^W × H (X(i,j,:) is equal to the zero vector if not sampled, i.e., corresponding to S(i,j) = 0), and a conventional RGB image I̅∈ℝ^W × H × b with the same spatial resolution as the target XRF image Y̅, but a small number (equal to 3) of spectral bands, i.e., b ≪ B. Scanned data X̅ and I̅ is linearly scaled to [0 1], i.e., 0 ≤X(i,j,k) ≤ 1 and 0 ≤I(i,j,k) ≤ 1. Note that the RGB image is fully sampled, and therefore the primary goal of the reconstruction algorithm is to transfer image information from the RGB image to regions of the XRF where no samples are acquired. The input subsampled XRF image X̅ can be separated into two parts: the visible component X̅_v ∈ℝ^W × H × B and the non-visible component X̅_nv∈ℝ^W × H × B, with the same binary sampling mask S̅ as X̅. We propose to estimate the fully sampled visible component Y̅∈ℝ^W × H × B by fusing the conventional RGB image I̅ with the visible component of the input subsampled XRF image X̅_v, and the fully sampled non-visible component Y̅_nv∈ℝ^W × H × B by using standard total variation inpainting methods.

To simplify notation, the image cubes are written as matrices, i.e., all pixels of an image are concatenated, such that every column of the matrix corresponds to the spectral response at a given pixel, and every row corresponds to a lexicographically ordered spectral band. Those unsampled pixels are skipped in this matrix representation. Accordingly, the image cubes are written as Y ∈ℝ^B × N_h, X ∈ℝ^B × N_s, I ∈ℝ^b × N_h, X_v ∈ℝ^B × N_s, X_nv∈ℝ^B × N_s, Y_v ∈ℝ^B × N_h, Y_nv∈ℝ^B × N_h, where N_h = W × H and N_s = W × H × c is the number of sampled XRF pixels. We therefore have


    X = X_v + X_nv,



    Y = Y_v + Y_nv,

according to the visible/non-visible component separation models as shown in Figure <ref>.

Let us denote by y ∈ℝ^B, y_v ∈ℝ^B, and y_nv∈ℝ^B the one-dimensional spectra at the same pixel location of Y̅, Y̅_v and Y̅_nv, respectively. That is, a column of Y_v and Y_nv is represented according to the linear mixing model <cit.> described as


    y_v = ∑_j=1^M d^xrf_v,jα_v,j,    Y_v = D_v^xrfA_v,



    y_nv = ∑_j=1^M d^xrf_nv,jα_nv,j,    Y_nv = D_nv^xrfA_nv,

where d^xrf_v,j and d^xrf_nv,j are column vectors representing respectively the endmembers for the visible and non-visible components, M is the total number of endmembers, D_v^xrf≡ [d^xrf_v 1, d^xrf_v 2,  …, d^xrf_v M] ∈ℝ^B × M, D_nv^xrf≡ [d^xrf_nv 1, d^xrf_nv 2,  …, d^xrf_nv M] ∈ℝ^B × M, and α_v,j and α_nv,j are the corresponding per-pixel abundances. Equation <ref> can be written per column, by utilizing the same column in each of the three matrices involved, that is y=y_v+y_nv. We take the corresponding α_v,j, j=1,...,M and stack them into an M × 1 column vector. This vector then becomes the k^th column of the matrix A_v ∈ℝ^M × N_h. In a similar manner, we construct matrix A_nv∈ℝ^M × N_h. The endmembers D^xrf_v and D^xrf_nv act as basis dictionaries representing Y_v and Y_nv in a lower-dimensional space ℝ^M, with rank{Y_v}≤ M, and rank{Y_nv}≤ M.

The visible X_v and non-visible X_nv components of the input subsampled XRF image are spatially subsampled versions of Y_v and Y_nv respectively; that is


    X_v=Y_vS=D^xrf_vA_vS,



    X_nv=Y_nvS=D^xrf_nvA_nvS,
 where S ∈ℝ^N_h × N_s is the subsampling operator that describes the spatial degradation from the fully sampled XRF image to the subsampled XRF image.

Similarly, the input RGB image I can be described by the linear mixing model <cit.>,


    I=D^rgb A_v,
 where D^rgb∈ℝ^b × M is the RGB dictionary. Notice that the same abundance matrix A_v is used in Equations <ref> and <ref>. This is because the visible component of the scanning object is captured by both the XRF and the conventional RGB images. Matrix A_v encompasses the spectral correlation between the visible component of the XRF and the RGB images.




 §.§ Proposed Solution



To solve the XRF image inpainting problem, we need to estimate A_v, A_nv, D^rgb, D^xrf_v and D^xrf_nv simultaneously. Utilizing Equations <ref>, <ref>, <ref>, and <ref>, we formulate the following constrained least-squares problem:




    min_A_v, A_nv, D^rgb,
     D^xrf_v,D^xrf_nv       X-D^xrf_vA_vS- D^xrf_nvA_nvS_F^2  
        +γ∇_I (D^xrf_vA_v) _F^2 + λ∇ (D^xrf_nvA_nv) _F^2 
        +I-D^rgbA_v_F^2 
    
    s.t.       0 ≤ D^xrf_v ij≤ 1, ∀ i,j 
        0 ≤ D^xrf_nv ij≤ 1, ∀ i,j 
        0 ≤ D^rgb_ij≤ 1, ∀ i,j 
        A_v ij≥ 0, ∀ i,j 
        A_nv ij≥ 0, ∀ i,j 
       1^𝐓(A_v+A_nv)=1^𝐓, 
        A_v + A_nv_0 ≤ s,


with ·_F denoting the Frobenius norm and ·_0 the ℓ_0 norm, i.e., the number of nonzero elements of the given matrix. D^xrf_v,ij, D^xrf_nv,ij, D^rgb_ij, A_v,ij, and A_nv,ij are the (i,j) elements of matrices D^xrf_v, D^xrf_nv, D^rgb, A_v, and A_nv respectively, and 1^𝐓 denotes a row vector of 1's compatible with the dimensions of A_v and A_nv. Equations <ref>b, <ref>c, and <ref>d enforce the nonnegative, bounded spectrum constraints on endmembers, Equations <ref>e and <ref>f enforce the nonnegative constraints on abundances, and Equation <ref>g enforces the constraint that the visible component abundances and non-visible component abundances for every pixel sum up to one. These physically grounded constraints from <cit.> are shown to be effective in our previous work <cit.>, by making full use of the fact that the XRF endmembers are XRF spectra of individual materials, and the abundances are proportions of those endmembers. 

The first term in Equation <ref>a represents a measure of the fidelity to the subsampled XRF data X, the second term is the total variation (TV) regularizer of the visible component, the third term is the TV regularizer of the non-visible component, and the last term is the fidelity to the observed RGB image I. The TV regularizer of the visible component ∇_I (D^xrf_vA_v) is defined as


    [                                                      ∇_I (D^xrf_vA_v) _F^2; = ∑_i=1^H-1∑_j=1^W-1 w_i,j^downD_v^xrfA̅_v(i,j,:)-D_v^xrfA̅_v(i+1,j,:)_2^2;                    +w_i,j^rightD_v^xrfA̅_v(i,j,:)-D_v^xrfA̅_v(i,j+1,:)_2^2;                                                   =  D_v^xrf A_v P(I)_F^2, ]

where A̅_v∈ℝ^W × H × M is the 3D volume version of A_v and A̅_v(i,j,:) ∈ℝ^M is the visible component abundance of pixel (i,j). w_i,j^down and w_i,j^right are the adaptive TV weights in the vertical and horizontal directions respectively; that is,


    w_i,j^down  = e^-αI̅(i,j,:) - I̅(i+1,j,:)_2^2,



    w_i,j^right = e^-αI̅(i,j,:) - I̅(i,j+1,:)_2^2,

where I̅(i,j,:) is the RBG image pixel at position (i,j). P(I) ∈ℝ^N_h × ((W-1)(H-1)) in Equation <ref> is the adaptive horizontal/vertical first order difference operator determined by the input RGB image I according to Equations <ref> and <ref>. Equations <ref> and <ref> indicate that the TV regularizer of the visible component adapts to the dense set of information that is available in the conventional RGB image I̅. When the difference between two adjacent RGB pixels is small, a strong spatial smoothness constraint is placed on their corresponding XRF pixels, and vice versa. This adaptive TV regularizer is one of the main differences between this fusion-based XRF image inpainting algorithm and our previous fusion-based XRF image SR algorithm <cit.>. We found out that such TV regularizer on the visible component is essential for the inpainting problem, otherwise the inpainting results are not satisfactory. For the SR approach, we do not need such a TV regularizer on the visible component. The SR degradation model assumes that the LR measured XRF image is a weighted sum of all the pixels in the target HR XRF image, so there is an implicit spatial smoothness constraint imposed by the LR XRF image. However, for the XRF image inpainting problem, we subsample the XRF image to obtain the measurement so that many pixels are not sampled at all, making the reconstruction more difficult than for the SR problem. 

The third term in Equation <ref> is a TV regularizer of the non-visible component. Its detailed definition can be found in our previous work <cit.> (see Equation (10)).

The optimization in Equation <ref> is non-convex and difficult to carry out if we are to optimize over all the parameters A_v, A_nv, D^rgb, D^xrf_v, and D^xrf_nv directly. Empirically we found out that it is effective to alternatively optimize over these parameters. Also, because Equation <ref> is highly non-convex, a good initialization is needed. Let Y^(0)∈ℝ^B × N_h be the initialization of Y. Such initialization can be obtained by utilizing some standard image inpainting algorithms <cit.> to inpaint the subsampled XRF image channel by channel. Then, the coupled dictionary learning technique in <cit.> can be utilized to initialize D^rgb and D^xrf_v by


    [                         min_ D^rgb, D_v^xrf I - D^rgbA_v_F^2 +  Y^(0) - D^xrf_v A_v_F^2;                                                               + β∑_k=1^N_l A_v(:,k) _1,;                                        s.t.                      D^rgb(:,k)_2 ≤ 1, ∀ k,;                                                                D^xrf_v(:,k)_2 ≤ 1, ∀ k,;                                             ]
 where ·_1 is the ℓ_1 vector norm, parameter β controls the sparseness of the coefficients in A_v, and A_v(:,k), D^rgb(:,k), and D^xrf_v(:,k) denote the k^th column of matrices A_v, D^rgb, and D^xrf_v respectively. Details of the optimization can be found in <cit.>. D^rgb and D_v^xrf are initialized using Equation <ref> and D_nv^xrf is initialized to be equal to D_v^xrf. A_v is initialized by Equation <ref> as well, while A_nv is set equal to zero at initialization. Note that this formulation allows our approach to naturally handle the problem when no hidden layers are present. The iterative optimization algorithms we deploy are similar to the ones used in our previous work <cit.>, and they are not repeated here.

Once the optimization problem in Equation <ref> is solved according to Equations <ref>, <ref>, and <ref>, the reconstructed fully sampled XRF output image Y can be computed by


    Y = Y_v + Y_nv=D_v^xrfA_v+D_nv^xrfA_nv.






§ EXPERIMENTAL RESULTS



In this section, we will first demonstrate the advantages of our developed adaptive sampling mask when applied to RGB image sampling and reconstruction. We will then demonstrate the performance of the proposed fusion-based XRF inpainting algorithm.




 §.§ Adaptive Sampling Mask for RGB Image Sampling and Reconstruction



For the RGB image sampling and reconstruction task, we show the benefits of our developed adaptive sampling mask, over the use of a random sampling mask, as well as other adaptive sampling algorithms, such as Adaptive Irregular Sampling (AIrS) proposed in <cit.>, Kernel-based Adaptive Sampling (KbAS) proposed in <cit.> and Measurement-Adaptive Sampling and Cellular Automaton Recovery (Mascar) proposed in <cit.>.



  §.§.§ Datasets



To train our proposed adaptive sampling mask generation CNN in Section <ref>,  the ImageNet <cit.> database without any of the accompanying labels is used. We randomly select 1,000,000 images for the training set, 1000 for the validation set, and 1000 for the testing set. All images are selected randomly among all categories to capture as diverse image structures as possible and are cropped to have spatial resolution 64 × 64 pixel.



  §.§.§ Implementation Details









Our proposed adaptive sampling mask generation CNN (Section <ref>) is implemented in PyTorch[<https://github.com/usstdqq/deep-adaptive-sampling-mask>]. ADAM <cit.> is applied as the stochastic gradient descent solver for optimization. We use the same hyper-parameters suggested in <cit.> and batch size equal to 128 during training. 400 epochs are applied during the training process. We test 3 sampling rates: c = 5%,  c = 10%, and  c = 20% for the sampling percentage parameter. A 5%, 10%, and 20% sampling percentage roughly speed up the raster scanning procedure by a factor of 20, 10, and 5, respectively.

For training, we first initialize the inpainting network NetE according to <cit.>. A random sampling mask with sampling rate c is utilized to sample the input RGB image. The mask generation network NetM is initialized randomly. We then train the whole network architecture in Figure <ref>. The learning rate of the mask generation network NetM is set equal to 0.0002 during training. The learning rate of the inpainting network NetE is set to 0, i.e., we fix NetE when training NetM. We did not optimize NetM and NetE simultaneously; although the best reconstruction of z would have been obtained, the two networks would have been dependent on each other. Notice that the channel-wise fully connected layer in NetE (Figure <ref>) is able to learn a high-level feature mapping, making NetE able to perform semantic image inpainting. However, we would like to utilize other image inpainting algorithms other than NetE to make the adaptive sampling mask be as general and applicable to as many image inpainting algorithms as possible. Also as mentioned in Section <ref>, during training, the output of NetM is D ∈𝐑^2, a probabilistic map, instead of the binary mask Ber(D). So training NetE with NetM jointly would make NetE learn to reconstruct images corrupted by multiplying them by a continuous probabilistic mask, instead of a binary mask, which does not represent the inpainting problem. By fixing NetE, which is pre-trained by random binary sampling masks, NetM is constrained to be optimized for the general image inpainting problem instead of the “probabilistic” image inpainting problem. An adaptive sampling mask is thus trained suitable for general image inpainting problem. Note that, since NetE architecture is differentiable, it allows us to train the NetM atchitecture to optimize the mask, while taking into account both the RGB image content, and the reconstruction algorithm. 







  §.§.§ Performance on ImageNet Testing Images




To compare the performance of our adaptive sampling mask (NetM) with the random, AIrs <cit.>, KbAS <cit.> and Mascar <cit.> sampling masks, we apply all of them to corrupt  1000 testing images from the ImageNet database. Three sampling rates  c = 0.05, c = 0.1 and c = 0.2 are tested for the sampling methods. For image inpainting algorithms, NetE Inpainting <cit.>, Harmonic Inpainting <cit.>, Mumford-Shah (Mum-Sh) Inpainting <cit.>, and BPFA inpainting <cit.> are used to reconstruct the fully sampled RGB images.









The average PSNRs in dB over all 1000 test images are shown in Table <ref>. First, we observe that under all three sampling rates, the proposed NetM mask outperforms the random, AIrS, KbAS and Mascar sampling masks consistently over all inpainting reconstruction algorithms in terms of PSNR, showing the effectiveness of our proposed adaptive sampling mask generation network. Furthermore, the proposed NetM is significantly faster than AIrS, KbAS, and Mascar in generating the sampling mask. Finally, it can be concluded that the smaller the sampling rate, the larger the advantage of our proposed algorithm compared to other sampling algorithms. This implies that our proposed NetM is able to handle challenging sampling tasks (small sampling rates) and have better reconstruction accuracy under various reconstruction algorithms.

The visual quality comparison of the adaptive sampling and random sampling masks is shown in Figure <ref>. Two test images are picked from the testing set of 1000 images in total. Under sampling rate c=0.2, the advantages of the proposed NetM mask over the random, AIrS, KbAS and Mascar sampling masks can be observed by comparing the resulting reconstructions by the same inpainting algorithm. For the test image #39, the NetM mask is able to capture the white dots in the red hat, resulting in accurate reconstruction results of those white dots. KbAS misses one white dot in the image. Although AIrS and Mascar are able to sample all the white dots in the image, they fail to capture the structure of these white dots. For test image #91, compared to random sampling mask, the proposed NetM mask samples the contour structure of the bird, resulting in its better reconstruction. When compared to AIrS, KbAS and Mascar masks, the proposed NetM mask samples the whole image more evenly, resulting in fewer artifacts in the inpainted images. The improved performance of the adaptive sampling mask over other sampling masks is consistent over all inpainting reconstruction algorithms we tested.



  §.§.§ Performance on Painting Images



We also tested our proposed adaptive sampling algorithm on painting images at sampling rate c=0.1. As shown in Figures <ref> (a), the RGB image of the painting “Bloemen en insecten” is tested. It has spatial resolution 580 × 680 pixels. Random, AIrS, KbAS, Mascar and NetM sampling masks are generated as shown in Figures <ref>(c), (g), (k), (o) and (s) with the corresponding computation time. Harmonic Inpainting <cit.>, Mumford-Shah Inpainting <cit.>, and BPFA <cit.> algorithms are utilized to reconstruct the sampled RGB images, and the reconstruction results are shown in Figures <ref> (d)-(f), (h)-(j), (l)-(n), (p)-(r) and (t)-(v) with the corresponding PSNR values. By comparing the rows of different sampling masks, it can be concluded that our proposed NetM mask outperforms other sampling masks in terms of both visual quality of the reconstructed images and the PSNR values. Notice that NetM is significantly faster than AIrS, KbAS and Mascar in computation speed. Although KbAS samples densely on the foreground, it still misses many details due to the complexity of the flower structure. Similarly, Mascar samples densely on the foreground, while it misses the flower stem structure. NetE Inpainting <cit.> is not utilized in this experiment since it is trained to inpaint RGB images with spatial resolution 64 × 64 pixels. The network structure shown in Figure <ref> is not fully convolutional, as there is the channel-wise fully connected layer in the middle. NetM is fully convolutional (Figure <ref>) so that it can generate sampling masks of input images with arbitrary resolution.




 §.§ Adaptive Sampling Mask for X-Ray Fluorescence Image Inpainting



In the previous section (Section <ref>), we demonstrated the effectiveness of our proposed NetM sampling mask on the RGB image sampling and inpainting problem. To further evaluate the effectiveness of the NetM sampling mask and evaluate the performance of our proposed fusion-based inpainting algorithm (Section <ref>), we have performed experiments on XRF images. The basic parameters of the proposed reconstruction method are set as follows: the number of atoms in the dictionaries D^rgb, D_nv^xrf and D_v^xrf is M=200; parameter λ and γ in Equation <ref> are set equal to 0.1; parameter α in Equation <ref> and Equation <ref> is set to 16. The optional constraint in Equation <ref>h is not applied here.




  §.§.§ Error Metrics



The root mean squared error (RMSE), the peak-signal-to-noise ratio (PSNR), and the  spectral angle mapper (SAM, <cit.>) between the estimated fully sampled XRF image Y and the ground truth image Y^gt are used as the error metrics. 




  §.§.§ Comparison Methods



According to our knowledge, no work has been reported on solving the XRF (or Hyperspectral) image inpainting problem by fusing a conventional RGB image. So, we can only compare our results with traditional image inpainting algorithms such as  Harmonic Inpainting <cit.>, Mumford-Shah Inpainting <cit.>, and BPFA inpainting <cit.>. Harmonic Inpainting and Mumford-Shah Inpainting methods are for image inpainting, so we have to inpaint the XRF image channel by channel. BPFA inpainting <cit.> is able to inpaint multiple channels simultaneously. For the sampling mask comparison, we still compare with random, AIrS <cit.>, KbAS <cit.> and Mascar <cit.> sampling masks.




  §.§.§ Real Experiment



For this experiment, real data were collected by a home-built X-ray fluorescence spectrometer (courtesy of Prof. Koen Janssens), with 2048 channels in spectrum. Studies from the XRF image scanned from Jan Davidsz. de Heem's “Bloemen en insecten” (ca 1645), in the collection of Koninklijk Museum voor Schone Kunsten (KMKSA) Antwerp, are presented here. We utilize the super-resolved XRF image in our previous work <cit.> as the ground truth. The ground truth XRF image has dimensions 680 × 580 × 2048. We first extract 20 regions of interest (ROI) spectrally and work on them, to decrease the spectral dimension from 2048 to 20. We decrease the spectral dimension so as to compare with other inpainting algorithms, e.g., <cit.>, which therefore reconstruct the subsampled XRF image channel by channel and large spectral dimensions will make the computational time very long. The sampling ratio c is set to be 0.05, 0.1, and 0.2. Different sampling strategies are applied and analyzed. Various inpainting methods are subsequently applied to reconstruct those subsampled XRF images.









As shown in Table <ref>, our proposed fusion-based inpainting algorithm with the proposed adaptive sampling mask provides the closest reconstruction to the ground truth XRF image compared to all other methods. Our proposed algorithm utilizes as guidance a conventional fully sampled and high contrast RGB image (Figure <ref> (a)), resulting in better inpainting performance. By comparing the difference between results by “Mum-Sh” and results by “Proposed” under various sampling rates, it can be concluded that the benefit gained by our proposed fusion-based inpainting is large when the adaptive sampling masks are applied. For example, at sampling rate c=0.2, there is a 0.78 dB improvement in PSNR by applying our proposed fusion-based inpainting algorithm when a random sampling masks are applied, while there is a  6.71 dB, 7.94 dB, 9.06 dB and 7.33 dB improvement in PSNR when an AIrS, KbAS, Mascar and NetM sampling mask is applied, respectively. This is because the adaptive sampling masks sampled the corresponding visible component of the XRF image efficiently and the fusion inpainting propagated the measured XRF pixels properly. Furthermore, the NetM sampling mask provides the best XRF image reconstruction compared to other sampling masks.

The iteration process of our proposed fusion inpainting algorithm when c=0.2 is shown in Figure <ref>. Notice that at the initial iterations of our proposed fusion inpainting algorithm, the RMSE is higher than the Mumford-Shah inpainting algorithm. This is because we decompose the inpainting result of Mumford-Shah inpainting algorithm by sparse representation, according to Equation <ref>. Due to the complexity of the “Bloemen en insecten” data, we lose some accuracy during the first few iterations. However, with more iterations, the RMSE of both random sampling and adaptive sampling decreases and becomes smaller than the RMSE of Mumford-Shah inpainting algorithm. Our proposed NetM sampling mask provides the smallest RMSE for both Mumford-Shah initialization and the proposed fusion inpainting algorithm, showing the effectiveness of NetM in the task of XRF image sampling and reconstruction. In addition, the optimization of Equation <ref> exhibits monotonic convergence under all sampling masks.


When c=0.2, the visual quality of the different inpainting algorithms and sampling strategies on channel #16, corresponding to the Pb Lη XRF emission line, is compared in Figure <ref>. The same random, AIrS, KbAS, Mascar and NetM sampling masks as the sampling masks in Figures <ref> (b), (f), (j), and (n), are applied here. The reconstruction results obtained by the Harmonic, Mum-Sh, BPFA, and Proposed algorithms using AIrS, KbAS, Mascar and NetM sampling masks are sharper than those using the random sampling mask. This is because the majority of the XRF signal in the “Bloemen en insecten” data correlates to the RGB signal, and the sampling masks, which are designed for the RGB image, would also be suitable for the visible component of the XRF signal. Among the AIrS, KbAS, Mascar and NetM sampling masks, NetM sampling mask outperforms others for different reconstruction algorithms, both quantitatively (Table <ref>) and qualitatively (Figure <ref>). The proposed fusion inpainting algorithm further improves the contrast and resolves more fine details in (q). When compared to the ground truth image (b), we can conclude that those resolved details have high fidelity to the ground truth image (b). 

Our proposed adaptive sampling algorithm NetM is not designed and trained for the task of XRF image sampling and reconstruction. However, NetM still outperforms random sampling, as well as three other adaptive sampling algorithms, under various XRF reconstruction algorithms and under different sampling rates. This illustrates the effectiveness of NetM in extracting image information with limited sampling budget. This also demonstrates that NetM can generalize well into some other imaging tasks.




  §.§.§ XRF Inpainting v.s. XRF Super-Resolution



In this experiment, we further compare the proposed XRF inpainting utilizing NetM adaptive sampling with our previous XRF-SR approach <cit.>. The XRF scanner can perform regular sub-sampling with a sampling step size K on both image dimensions. An upscale factor K SR can be applied to reconstruct the fully sampled XRF images. We set K=5 to match the SR setting of<cit.>. We retrain NetM with sampling rate c=0.04, which has the same number of samples as a K=5 regular sub-sampling. The training procedure is the same as described in Section <ref>.





As shown in Table <ref>, as expected, the proposed XRF inpainting algorithm utilizing NetM outperforms the XRF-SR approach. Please notice that both methods fuse an RGB image during the XRF reconstruction, and the main difference is the sampling pattern. This experiment further argues the improved effectiveness of the proposed NetM based sampling.



  §.§.§ Simultaneous Training of NetE and NetM



With an initial look at Fig. <ref>, it would seem meaningful to train NetE and NetM simultaneously and not sequentially, as described in Section <ref>. In this experiment, we performed certain experiments to test such approach and eventually support our proposed sequential training procedure. There are two approaches in training NetE and NetM simultaneously. The first one is to train NetE individually first, and then train NetE with NetM jointly using Equation <ref>. The second one is to perform random initialization on both NetE and NetM, and then simultaneously train them using Equation <ref>. We tested both of these training approaches at sampling rate c=0.1. We henceforth refer to the first approach as NetM_pre_train_joint, to the second approach as NetM_rand_init_joint and to the proposed training approach as NetM_sequential.  Notice that we quantize the probabilistic map D generated by NetM during testing.

We tested these two above described networks utilizing the same 1000 ImageNet test set and compared them with the other inpainting approaches considered in this paper in terms of PSNR (in dB). As is clear from Table <ref>, the results obtained by the networks NetM_pre_train_joint  and NetM_rand_init_joint , with which NetM and NetE were trained simultaneously, are not competitive with the results obtained by the proposed sequential training of NetE and NetM, i.e., NetM_sequential. It is mention again here that NetE in both of these simultaneous training approaches was trained with input images corrupted by a probabilistic map D. The resulting sampling masks from these two approaches cannot also get good reconstruction accuracy using Harmonic <cit.> and Mum-Sh <cit.> inpainting. There is approximately a 2 dB difference in reconstruction accuracy, indicating that the trained NetM is not as effective. When comparing these two simultaneous optimization approaches to each other, it is observed that using the pre-trained NetE before joint training provides some benefit on both NetE and NetM.





§ CONCLUSION



In this paper, we presented a novel adaptive sampling mask generation algorithm based on CNNs and a novel XRF image inpainting framework based on fusing a conventional RGB image. For the adaptive sampling mask generation, we trained the mask generation network NetM along with the inpainting network NetE to obtain an optimal binary sampling mask based on the input RGB image. For the fusion-based XRF image inpainting algorithm, the XRF spectrum of each pixel is represented by an endmember dictionary, as well as the RGB spectrum. The input subsampled XRF image is decomposed into visible and non-visible components, making it possible to find the nonlinear mapping from the RGB to the XRF spectrum. Experiments show the effectiveness of our proposed network NetM in both RGB and XRF image sampling and reconstruction tasks. Higher reconstruction accuracy is achieved in both RGB and XRF image sampling, and also the computation time of mask generation is significantly smaller than other adaptive sampling methods.




































































































































































IEEEtran





















































