Abstract
were developed to detect the presences of multiple gasses in near infrared (IR) spectrums. High Resolution Transmission database was used to synthesize the near IR spectrums. Monte Carlo KNN determined the optimal kernel sizes and the optimal number of channels. The multi-resolution CNN, composed of multiple different kernels, was created using the optimal kernel sizes and the optimal number of channels. The multiresolution CNN outperforms the multilayer perceptron and the partial least squares. © Deep learning has brought many improvements to the field of spectroscopy. Afara et al. [12] used deep learning for the classification of cartilage integrity in near IR spectroscopy. Gan et al. [13] used PCA with a multilayer perceptron (MLP) for classifying multiple gasses in near IR spectrums. Furthermore, Gan et al. [13] used optimal thresholding to balance the false positive errors and the false negative errors in order to obtain the best performance. Chen et al. [14] implemented convolutional neural networks (CNNs) for hyperspectral image classification. The paper [14] implemented 2x2, 4x4, and 5x5 kernels. However, there are multiple challenges in the selection of the kernel sizes, the kernel strides, and the number of channels. The kernel sizes were manually found by executing a random search or a grid search. The manual searches require a lot of time as the CNN needs to be retrained for every change in the CNN hyper-parameters. Furthermore, the process for selecting a new kernel size estimate is arbitrary. Manual searches are not well guided. Moreover, manual searches do not guarantee the best performance as manual searches might only lead to a local optima.
Abstract
As a result of these challenges, a Monte Carlo k-nearest neighbours (KNN) and a multi-resolution CNN were developed for multi-gas spectrum classification. The Monte Carlo KNN solves the CNN hyper-parameter tuning problem by selecting the optimal kernel sizes, optimal number of channels, and optimal kernel strides. The multi-resolution CNN was built using the optimal hyper-parameters in order to extract features from the near IR spectrums. After extracting the features from the spectrums, an MLP was developed to classify the presences of the gasses in the spectrums. KNN [15] is an unsupervised clustering algorithm. KNN clusters data points together based on euclidean distance. If two data points are close enough, then KNN will assign both of the data points to the same cluster. If two data points are far away from each other, then KNN will assign the data points to different clusters. As a consequence, the KNN enables the clustering of similar data points based on euclidean distance.
Abstract
Algorithm 1 shows the Monte Carlo KNN algorithm. The Monte Carlo KNN determines the optimal hyper-parameters for the multi-resolution CNN. Firstly, windows of random lengths

http://dx.doi.org/10.1364/ao.XX.XXXXXX

The field of spectroscopy excels at the detection of elements or chemical compounds in unknown substances. For example, infrared (IR) absorbance spectroscopy allowed Al-Tameme et al. [1] to determine the chemical composition of Urtica dioica leaves. Zaini et al. [2] determined the chemical concentrations of CO 3 and Al OH in cement-grade limestone using IR absorbance spectroscopy. Wang et al.'s paper [3] used near IR absorbance spectroscopy for the analysis of alcoholic beverages, non-alcoholic beverages, milk, and oils. Furthermore, Wang et al. [3] classified the types of liquid foods as well as determined the quality of the liquid foods. Ryde et al. [4] estimated the concentrations of H, Fe, Si, Ti, Ca, Mg, and Sc of a red giant using multiple emission spectroscopy bands.

Fourier transforms are popular in the field of spectroscopy. Yang et al. [5] analyzed IR spectrums using the Fourier transform. Spectroscopy allowed Yang et al. [5] to retrieve data about the protein structures in aqueous solutions. Fuller and Ogilvie [6] created 2D Fourier transform algorithms for the analysis of spectrums. Nesakumar et al. [7] used the Fourier transform and the principal component analysis (PCA) for the detection of moisture content in beetroot.

Generic algorithms evolve parameters for the classification of chemical compounds in absorbance spectrums. Moreover, partial least squares (PLS) [8] utilizes matrix decomposition to extract features from the absorbance spectrums. The extracted features excel at predicting the presences of chemical compounds. For example, Xie et al. [9] used genetic algorithms and PLS to analyze near IR spectrums. The goal of Xie et al.'s paper [9] was to classify the types of soils using near IR spectrums. Yin et al. [10] applied genetic algorithms and PLS to terahertz spectroscopy. Yin et al. [10] wanted to identify the types of edible oils in foods. Zhu et al. [11] used genetic algorithms and PLS to detect the emissions of boron monoxide.

Deep learning has brought many improvements to the field of spectroscopy. Afara et al. [12] used deep learning for the classification of cartilage integrity in near IR spectroscopy. Gan et al. [13] used PCA with a multilayer perceptron (MLP) for classifying multiple gasses in near IR spectrums. Furthermore, Gan et al. [13] used optimal thresholding to balance the false positive errors and the false negative errors in order to obtain the best performance. Chen et al. [14] implemented convolutional neural networks (CNNs) for hyperspectral image classification. The paper [14] implemented 2x2, 4x4, and 5x5 kernels. However, there are multiple challenges in the selection of the kernel sizes, the kernel strides, and the number of channels. The kernel sizes were manually found by executing a random search or a grid search. The manual searches require a lot of time as the CNN needs to be retrained for every change in the CNN hyper-parameters. Furthermore, the process for selecting a new kernel size estimate is arbitrary. Manual searches are not well guided. Moreover, manual searches do not guarantee the best performance as manual searches might only lead to a local optima.

As a result of these challenges, a Monte Carlo k-nearest neighbours (KNN) and a multi-resolution CNN were developed for multi-gas spectrum classification. The Monte Carlo KNN solves the CNN hyper-parameter tuning problem by selecting the optimal kernel sizes, optimal number of channels, and optimal kernel strides. The multi-resolution CNN was built using the optimal hyper-parameters in order to extract features from the near IR spectrums. After extracting the features from the spectrums, an MLP was developed to classify the presences of the gasses in the spectrums.

KNN [15] is an unsupervised clustering algorithm. KNN clusters data points together based on euclidean distance. If two data points are close enough, then KNN will assign both of the data points to the same cluster. If two data points are far away from each other, then KNN will assign the data points to different clusters. As a consequence, the KNN enables the clustering of similar data points based on euclidean distance.

Algorithm 1 shows the Monte Carlo KNN algorithm. The Monte Carlo KNN determines the optimal hyper-parameters for the multi-resolution CNN. Firstly, windows of random lengths Algorithm 1. Monte Carlo KNN 1: procedure MONTECARLO(spec) 2: spec_len ← Length of spec 3: f ilter_list ← Empty list 4: for i ← 0 to 100spec_len do 5: window_len ← Random number from 1 to spec_len 6: start_window ← Random number from 0 to spec_len − window_len 7: mean ← Mean of spec[start_window : window_len]

Append start_window, window_len, mean to f ilter_list 9: f ilter_list ← Highest means of f ilter_list 10: return f ilter_list 11: procedure MCKNN(spectrums) 12: kernel_list ← Empty list 13: for i ← 0 to number of spectrums do 14: Append MonteCarlo(spectrums[i]) to kernel_list 15: Perform KNN clustering on kernel_list to get the optimal kernel sizes and optimal number of channels 16: optimal_kernel_sizes ← Means of the clusters 17: optimal_channel_nums ← Number of points in a cluster 18: return optimal_kernel_sizes , optimal_channel_nums and random positions were generated and were placed on to the spectrums. Secondly, the means of the windows were taken. Thirdly, the windows with the highest means were selected as the kernel sizes. Fourthly, KNN [15] clustering was performed on kernel sizes as shown in Fig. 1 . Each kernel size was assigned to the closest cluster. For each cluster, the mean of the cluster becomes the optimal kernel size and the number of points in the cluster becomes the optimal number of channels. Finally, the kernel strides were determined by dividing the kernel sizes by 4 and being an integer divisor of the number of pixels in the spectrum. The optimal hyper-parameters allow the detections of specific features in the gas spectrums. The optimal hyperparameters for the CNN were shown in Table 1 .

Let LeakyReLU(x) = LeakyReLU activation function Fig. 2 . Architecture of the multi-resolution CNN.

Let X k = data input element of column k Let W k = kernel weight element of column k Let V z = 1D convolution element of column z Let M = number of columns in the kernel matrix Let B = bias matrix Let O = output of the 1D CNN

The multi-resolution CNN, shown in Fig. 2 , was constructed using the optimal hyper-parameters determined by the Monte Carlo KNN. Eqn. 1 to Eqn. 4 [16] show the equations for the multi-resolution CNN. The CNN kernels perform convolutions on the spectrum data in order to extract the spatial features of the spectrums. Features such as the spectral widths and the spectral peaks were extracted. The features were used to identify the individual gas spectrums. The multi-resolution CNN used 1D kernel sizes of 250 pixels, 800 pixels, 1200 pixels, 1500 pixels, and 4000 pixels. The CNN kernels used the LeakyReLU activation function α = 0.01. The LeakyReLU activation function was chosen to prevent the gradient from reaching zero. Batch normalization was applied to the CNN in order to decrease the training time of the multi-resolution CNN.

Let X i = data input vector of layer i Let W i = weight matrix of the layer i Let B i = bias vector of the layer i Let O i = layer output vector of layer i

Let J(ŷ, y) = cross-entropy loss function Let S(x) = sigmoid activation function with respect to x. Let y = gas presence label Letŷ = gas presence prediction

The MLP takes in the features from the multiple CNN kernels and the executes gas classification. Eqn. 5 [17] shows the equation for the MLP. The MLP has 3 layers. Each layer has 600 neurons and uses the LeakyReLU activation function α = 0.01. Furthermore, batch normalization was applied to the MLP. Eqn. 6 shows the cross-entropy loss function used to train the neural network. If the gas presence predictions do not match the gas presence labels, then the loss is large. If the gas presence predictions match the gas presence labels, then the loss is small. The MLP also outputs the final gas classification. If a gas is present, then the MLP predictsŷ = 1.0. If a gas is not present, then the MLP predictsŷ = 0.0.

Let OptimalThresholding(x) = optimal binary thresholding for classification with respect to x Let thres = optimal threshold for classification

Gan et al. [13] developed optimal thresholding for MLPs. Given an MLP prediction, the optimal threshold thres could be selected to maximize the F 1 score. Eqn. 7 shows the optimal thresholding equation. The maximum F 1 score occurs when precision approximately equals recall P ≈ R. A linear search was performed on the cross-validation dataset in order to find the optimal threshold thres. Moreover, the linear threshold search was performed independently for each gas.

Let TP = true positives Let FP = false positives Let FN = false negatives Let R = recall Let P = precision Let Fig. 3 . Learning curve of the multi-resolution CNN. 20 dB SNR. Fig. 3 shows the learning curve of the multi-resolution CNN. As the number of training samples increases, the cross-validation F 1 score increases until the F 1 score converges with the training F 1 score. When both F 1 scores converge, the F 1 scores prove the training convergence of the multi-resolution CNN. The optimal number of training samples is 300,000 spectrum samples.

The proposed solution was tested against PLS-BR [8] , PCA + MLP [13] , and a random CNN using the High Resolution Transmission (HITRAN) database [18] . The random CNN used randomly chosen kernel sizes. The random CNN was created to test the optimal selection of the kernel sizes. Firstly, 300,000 training samples were randomly selected from a pool of 450,000 samples. Furthermore, 20,000 cross-validation samples and 20,000 testing samples were selected from a different pool of 100,000 samples. Secondly, the algorithms were trained using the 300,000 training samples. Thirdly, the optimal thresholds were obtained using the 20,000 cross-validation samples. Fourthly, the algorithms were tested using the optimal thresholds on the 20,000 testing samples. Finally, the process above repeats 20 times for 1x20 fold testing. Fig. 6 show the algorithm test results. Precision, recall, and F 1 score were calculated using Eqn. 8 to Eqn. 10. For every dataset, the proposed solution performed better than the PLS-BR [8] , PCA + MLP [13] , and a random CNN. The proposed solution has a higher mean F 1 score than the rest of the algorithms. Moreover, the mean F 1 score of the proposed solution is 1 standard deviation higher than the mean F 1 score of the random CNN and the PCA + MLP [13] . The performance of the proposed solution is due to the optimal kernel size selection. The selected kernel sizes were tailored for classifying each individual gas spectrum.

In conclusion, a Monte Carlo KNN was created to select the optimal kernel sizes, optimal number of channels, and optimal kernel strides. The Monte Carlo KNN generated random windows of various sizes and positions. The windows that match the gas spectrums were selected as the kernel sizes. The multiresolution CNN used the optimal hyper-parameters from the Monte Carlo KNN. The multi-resolution CNN used multiple different kernels to extract features from the gas spectrums. The MLP layers used the features from the multi-resolution CNN to classify the gas presences. The proposed solution outperforms the PLS-BR [8] , PCA + MLP [13] , and a random CNN. Fig. 4 shows the proposed solution having a higher mean F 1 score than the rest of the algorithms. The HITRAN [18] database was used to synthesize the training, cross-validation, and testing datasets. Firstly, these individual gas spectrums were selected from HITRAN: C 2 H 6 , CH 4 , CO, H 2 O, HBr, HCl, HF, N 2 O, and NO. Secondly, the gas spectrums were constrained to have a wavelength range from 1µm to 7µm. The gas spectrums were uniformly down sampled to 10,000 pixels. Thirdly, a gas concentration matrix was randomly generated using a uniform distribution of concentrations ranging from 0µM to 10µM. Fourthly, each gas concentration value in the gas concentration matrix was randomly set to 0µM with a probability of p = 0.5. Fifthly, the gas spectrums were scaled using the gas concentration matrix. The gas spectrum scaling was done using the Beer-Lambert law shown in Eqn. 11. The gas concentrations c were taken from the gas concentration matrix and the path length was set to l = 10cm. The individual gas spectrums were also combined. Each training sample has 9 gas spectrums combined. Finally, white Gaussian noise was added to the synthesized gas spectrums.
APPENDIX


{'paper_id': '195798661', 'title': 'Classifying Multi-Gas Spectrums using Monte Carlo KNN and Multi-Resolution CNN', 's2_url': 'https://api.semanticscholar.org/CorpusID:195798661', 'source': 'S2ORC'}