Can Image Enhancement be Beneficial to Find Smoke Images in Laparoscopic Surgery?
Congcong Wang Vivek Sharma Yu Fan Faouzi Alaya Cheikh Azeddine Beghdadi Ole Jacob Elle Rainer Stiefelhagen
 

Quantum Adiabatic Algorithm Design using Reinforcement Learning
    Xiaopeng Li
    December 30, 2023
===============================================================



empty 







Laparoscopic surgery has a limited field of view. Laser ablation in a laproscopic surgery causes smoke, which inevitably influences the surgeon's visibility. Therefore, it is of vital importance to remove the smoke, such that a clear visualization is possible. In order to employ a desmoking technique, one needs to know beforehand if the image contains smoke or not, to this date, there exists no accurate method that could classify the smoke/non-smoke images completely. In this work, we propose a new  enhancement method which enhances the informative details in the RGB images for discrimination of smoke/non-smoke images. Our proposed method utilizes weighted least squares optimization framework (WLS). For feature extraction, we use statistical features based on bivariate histogram distribution of gradient magnitude (GM) and Laplacian of Gaussian (LoG). We then train a SVM classifier with binary smoke/non-smoke classification task. We demonstrate the effectiveness of our method on Cholec80 dataset. Experiments using our proposed enhancement method show promising results with improvements of 4% in accuracy and 4% in F1-Score over the baseline performance of RGB images. In addition, our approach improves over the saturation histogram based classification methodologies Saturation Analysis (SAN) and Saturation Peak Analysis (SPA) by 1/5% and 1/6% in accuracy/F1-Score metrics.

We can employ our enhancement method in replacement of RGB images for classifier training e.g., CNN architectures, which in turn can lead to more accurate classification. Code  will be released for public use.




§ KEYWORDS

Image Enhancement, Weighted Least Squares Framework (WLS), Smoke/Non-Smoke Image Classification.





§ INTRODUCTION
 
^⋆ Congcong Wang  and  Vivek  Sharma  contributed  equally  to  this  work  and
listed in alphabetical order.
Over the last decade, we have seen an increase in the number of laparoscopic surgeries <cit.>. During the surgery, such as in cavitary treatment, laser ablation causes smoke <cit.> which significantly degrades the perceptual quality of the images which inevitably influences the surgeon's visibility, further it also influences the performance of computer vision based navigation systems <cit.>. Moreover, surgical smoke is composed of chemical, physical or biological particles, which may be harmful for surgeons and patients <cit.>. Therefore, it is of vital importance to remove the smoke by computer vision algorithms <cit.> and by smoke evacuation techniques <cit.>.  In order to employ a desmoking technique, as a prior knowledge it is essential to know if the image contains smoke or not. In this work, we propose a method to enhance the images for better classification of smoke and non-smoke images.  Our goal is to enhance the images, such that the extracted features from the enhanced images are informative for discrimination that can lead to improved smoke/non-smoke image classification. Note that our goal is not to enhance the images for visual pleasantness of observers' perception, but rather enhance the images features for improved classification.

Our work is inspired from <cit.>. Sharma et al. in <cit.> enhance the visible (RGB) images using near-infrared (NIR) counterparts and show improvement in the image feature quality for biometric verification tasks, further in <cit.>, Sharma et al. emulates several image enhancement methods in convolutional neural networks for an accurate image classification. We have a similar goal, though our work differs substantially in technical approach and the application scope. Specifically, we utilize weighted least squares optimization framework (WLS) <cit.> to decompose an image to fine and coarse enhanced images, and then combine them in a more meaningful way such that the combined image have better image features for our classification task.

Our proposed approach is evaluated on Cholec80 dataset for smoke/non-smoke image classification. We experimentally show that our proposed method consistently improve the classification performance over the baseline RGB images, popular state-of-the-art enhancement methods, and the saturation histogram based classification methodologies Saturation Analysis (SAN) and Saturation Peak Analysis (SPA).

The remainder of this paper is structured as follows. First, we review the related work on image enhancement and smoke detection methods. Next, we describe our proposed method, and discuss the experimental results. Finally, the conclusions are drawn.





§ RELATED WORK
 



  
Image Enhancement.
Image enhancement or  filtering techniques enhance the contrast, boost the image details, and produce more vivid colors, and at the same time removes the effects of blur, noise, and compression artifacts. Examples of such filtering methods include weighted least squares (WLS) <cit.>, bilateral filtering <cit.>, image sharpening, guided filtering <cit.>, BFWLS_AVG <cit.> and more.   Filtering using (1) RGB and (2) RGB-NIR  images are used for several applications  in computer vision and computational photography applications, such as to improve the contrast of the haze-degraded color images <cit.>; tone mapping and detail enhancement <cit.>; denoising in RGB videos <cit.> and images <cit.>; multi-modal medical image fusion from MRI-CT <cit.>; illumination transfer from reference to target images <cit.>; feature matching <cit.>;  object recognition and image classification tasks <cit.>; biometrics verification tasks <cit.>; and the list goes on. To the best of our knowledge, our work is the first to show that the image enhancement can be beneficial for smoke/non-smoke image classification. We compare against various of these filtering methods discussed above in our experimental section.


Image enhancement or  filtering techniques enhance the contrast, boost the image details, and produce more vivid colors, and at the same time removes the effects of blur, noise, and compression artifacts. Examples of such filtering methods include weighted least squares (WLS) <cit.>, bilateral filtering <cit.>, image sharpening, guided filtering <cit.>, BFWLS_AVG <cit.> and more.   Filtering using (1) RGB and (2) RGB-NIR  images are used for several applications  in computer vision and computational photography applications, such as to improve the contrast of the haze-degraded color images <cit.>; tone mapping and detail enhancement <cit.>; denoising in RGB videos <cit.> and images <cit.>; reduce the contrast of the HDR images to display it on LDR media <cit.>; image restoration <cit.>; automatic skin enhancement <cit.>; flash/no-flash denoising <cit.>; realistic skin smoothing <cit.>; shadow detection <cit.>; multi-modal medical image fusion from MRI-CT <cit.>; illumination transfer from reference to target images <cit.>; image upsampling <cit.>; feature matching <cit.>;  object recognition and image classification tasks <cit.>; biometrics verification tasks <cit.>; and the list goes on. To the best of our knowledge, our work is the first to show that the image enhancement can be beneficial for smoke/non-smoke image classification. We compare against various of these filtering methods discussed above in our experimental section.




  
Smoke Detection.
Considerable progress has been seen in the development of “in the wild" video smoke detection techniques over the last decade <cit.>. The traditional detection methods exploit smoke features, which are intensity, color, motion, and texture  attributes. These features are used to train classifiers for smoke region detection <cit.> or frame detection <cit.>. It has been found that smoke can reduce the sharpness of edges in order to mitigate over it, several descriptors have been utilized for feature extraction, such as wavelet coefficients <cit.>, local binary pattern (LBP) <cit.>, textural features are estimated from the smoke region only <cit.>. Further, in <cit.>, the authors propose to separate smoke and background from a single image by the dual-dictionary approach, and the estimated sparse coefficients are used as features for smoke frames detection. 

In the medical imaging community, researchers have shown that the formation of smoke and the lighting condition for laparoscopic surgery images are very different, which limits the “in the wild" smoke detection methodologies applicability in medical domain <cit.>. Recently, Chou et al. <cit.> explore temporal differences information that is motion blur and block analysis between current and previous frames as features for smoke detection. Loukas et al. <cit.>,  propose a method to detect electrocautery smoke for surgical events retrieval. They extract different features from optical flow estimated by Kanade-Lucas-Tomasi (KLT) algorithm <cit.>, and then a support vector machine (SVM) is trained to classify each shot. Leibetseder et al. in <cit.> propose saturation histogram based classification methodologies Saturation Peak Analysis (SPA) and Saturation Analysis (SAN) and deep learning (DL) techniques.  In <cit.>, Leibetseder et al. train  three variants of convolutional neural network (CNN) models: GoogleLeNet <cit.> 
trained with RGB images (GLN RGB) and saturation channels only (GLN SAT), and a modified AlexNet <cit.> trained on RGB images (ALEX RGB). The performance of their CNN models on Cholec80 <cit.> dataset are very similar to that of SAN or SPA. While DL shows its promising performance on smoke classification <cit.>, to the best of our knowledge there is no work that does image enhancement for smoke classification. Therefore, we limit to comparisons with the non deep learning based methods only in our experimental section. Moreover, one can consistently improve the classification performance by employing our enhancement method and then train the CNN architectures, following the ideas proposed in Sharma et al. <cit.>. 









§ PROPOSED METHOD
 

In this section, we illustrate our proposed approach, starting with the proposed enhancement method, then the feature extraction method, and finally classifier training. Figure <ref> shows the schematic layout of our framework.







  
Proposed Enhancement Method. We denote our WLS-based filtering method as FC. To  enhance the visible RGB image, we first transform color space from RGB to YCbCr space that is a luminance-chrominance color space. Our method operates on the luminance component <cit.> such that to modify the overall contrast and sharpness of the image without sensibly affecting the color. Where the chrominance is simply re-combined in the final enhanced image.

In our work, we employ an edge-preserving filter, weighted least squares optimization framework (WLS) <cit.>. The WLS is a non-linear method that captures image details at a variety of scales via multi-scale decompositions. The WLS helps to find an approximate enhanced image g^Filtered that is close to the input image g, and also at the same time, is smooth along significant gradients, thus resulting to sharper preserved edges. Formally, it is defined as:

    g^Filtered = F_λ(g) = (I +  λ L_g)^-1g

where L_g = D_x^T A_xD_x + D_y^T A_yD_y with D_x and D_y are discrete differentiation operators.  A_x and A_y contain the smoothness weights, the smoothness requirement is enforced in a spatially varying manner which depend on g. λ is the balance factor that maintains a balance between the data term and the smoothness term. Increasing λ value  produces progressively smoother images. 

Given an input image, the WLS filter decomposes an image into base and detail layers. The detail layer is simply obtained by subtracting the base layer from the the original image. The base layer comprises of low frequency contents with general appearance of the image over smooth areas, while the detail layer comprises of high frequency contents with sharp edges. 

In our method, we apply WLS-based two-level decomposition of the luminance component of an RGB image for extraction of fine and coarse enhanced sharp images.  We retain, for each pixel an average value between the fine and coarse detail layers (Step 1). We chose λ_1=0.125 (WLS_1), λ_2=0.5 (WLS_2) for WLS in our experiments. 
The fusion criteria is based on the following observations: the WLS filter is very good at preserving fine and coarse details at arbitrary scales. Taking an average of two, allows to retain the informative content from both, thus allowing to preserve the image structures and also moderately boost the image details. This fusion criteria is denoted as FC_AVG. In addition, we also tried to retain the maximum values between the two as a fusion criterion. We denote this fusion criteria as FC_MAX. Finally, we combine the new fused detail layer with the base layer of RGB image obtained using WLS_1 (Step 2), we consider WLS_1, as smaller λ values are better for the base layer because of less smoothing. And which is then re-combined with the chrominance of RGB image and to reconstruct the final enhanced image (Step 3). Figure <ref> illustrates our proposed method.





  
Feature Extraction. In our work, we exploit Xue et al.'s method <cit.> for feature extraction. Xue et al. utilize gradient magnitude (GM) and Laplacian of Gaussian (LoG) maps to describe the structural information for image perceptual quality assessment. Their method is driven by the image statistics, and exploits the histogram information, which is perfect for low-level vision tasks, such as ours. Motivated by this observation, we employ GM and LoG features to represent the local spatial contrast information in images. 

Formally, gradient magnitude (GM) is defined as:

    GM=√((I⊗∂ G/∂ x)^2+(I⊗∂ G/∂ y)^2),

where I is the gray scale image, ⊗ denotes the convolution operation. ∂ G/∂ x and ∂ G/∂ y denote the Gaussian partial derivative along x (horizontal) and y (vertical) directions, respectively. They are computed as:


    ∂ G(x,y,σ )/∂ d=-1/2πσ^2d/σ^2e^-x^2+y^2/2σ^2,

where d ∈{x,y}, and σ is a scalar parameter. And the Laplacian of Gaussian (LoG) is defined  as:

    LoG=I⊗ h_LoG


    h_LoG(x,y,σ)   =∂ ^2 G(x,y,σ )/∂ ^2 x+∂ ^2 G(x,y,σ )/∂ ^2 y
       =-1/πσ^4(1-x^2+y^2/2σ^2)e^-x^2+y^2/2σ^2.

Subsequently, a joint adaptive normalization step is applied to adjust the image statistics of the GM and LoG maps, in order to reduce its dependency on local image content. Finally, a bivariate histogram based feature vector is computed from the marginal probability functions and independency distributions of the normalized GM and LoG maps, resulting to a fixed-size feature representation vector of 40 dimensions for an image. We used the default parameters for feature extraction, more details can be found in <cit.>.



  
Classifier. In our work, we use linear SVM <cit.> [] for a binary classification task in which the objective is to predict the class y ∈{0, 1}  that is  smoke/non-smoke image classification.  We use linear kernel function with C=10k to train/test SVM classifier with the extracted features. 







§ EXPERIMENTS
 

In this section, we demonstrate the use of our proposed enhancement method on a challenging smoke/non-smoke classification dataset <cit.>. We first introduce the dataset, followed by a thorough analysis of the proposed method. The analysis includes, comparison of our proposed method with baseline RGB images, other popular enhancement methods, and ending with a comparison to saturation histogram based classification methodologies. 










  
Dataset. We conduct experiments on Cholec80 dataset <cit.> which contains 80 videos of cholecystectomy surgeries manually labeled with smoke/non-smoke image sequence by <cit.> []. The dataset in overall contains approximately 100K annotated images, in particular between 200-1300 images of smoke/non-smoke in each video. In our experiments, we use a subset of the dataset with three videos for training and nine videos for testing. The videos are randomly chosen. We extract JPEG images, and resize them to a resolution of 427×240px from the original resolution of 854×480px, to fasten the image enhancement. In particular, we use 4,381 images obtained from video{1, 31, 34} for training, and 10,653 images obtained from video{2, 22, 10, 40, 59, 64, 65, 71, 80} for testing. In Figure <ref>, we show some examples of smoke/non-smoke images obtained from Cholec80 dataset. Also, note that each video represents a unique person in this dataset.



  
Evaluation Metrics. We use accuracy and F1-Score measures computed from a confusion matrix between the predicted labels and the actual ground-truth labels, as the metrics to evaluate the quality of classification.



  
Comparison with other enhancement methods.
We compare the proposed enhancement method with the baseline RGB images, and popular state-of-the-art enhancement approaches: (1) guided filtering (GF) <cit.>, (2) bilateral filtering (BF) <cit.>, (3) image sharpening filter (IMSHARP), (4) WLS <cit.>, (5) fused BF and WLS filter (BFWLS_AVG) <cit.>. In BFWLS_AVG, Sharma et al. <cit.> apply BF and WLS filters on the NIR channel, and retain an average of two for each pixel, similar to <cit.>, instead we employ their technique in the RGB image. For a fair comparison, we compare all the methods under the same evaluation protocol discussed above. For the evaluation, we use the same parameters for feature extraction and classifier training for all methods. We did not optimize the parameters for the BF, GF, WLS, IMSHARP, and BFWLS_AVG and used the default parameters for each method. The source code for fast BF [], WLS [] and BFWLS_AVG [] are publicly available, and others are available in the Matlab framework. For comprehensive discussion of different methods, we refer the reader to <cit.>.


In Table <ref>, we quantitatively evaluate the accuracy and F1-Score of our proposed method and other methods. We can clearly observe that  FC_AVG performs the best among all methods. Enhancing an image using FC_AVG takes approximately 0.32 seconds. Note that, FC_AVG improves over the baseline performance of RGB images by  4% in accuracy and 4% in F1-Score, giving higher-quality features to learn from, which in turn lead to more accurate classification. We believe our work opens many possibilities for further exploration for its usage in other tasks too. Further the performance gap of FC_AVG is 4/5% better than FC_MAX in accuracy/F1-Score measures. Our method FC_AVG consistently performs better than all other methods: IMSHARP, BF, WLS, and GF enhancement methods. In addition, our methods also performs better than BFWLS_AVG, although we agree that BFWLS_AVG is meant for RGB-NIR fusion and not RGB image enhancement, that may be the reason why it underperforms to our method. Figure <ref> (a) shows the ROC curve for all the methods. 

In Figure <ref>, we compare enhancement methods for an example image obtained from video1. We can observe that FC_AVG has improved features for the smoke-part (highlighted by red rectangles), in addition to reduced specular intensity when compared to FC_MAX. We believe this plays an important role in discrimination.










  
Comparison with the saturation histogram based classification methodologies.
We compare FC_AVG with the saturation histogram based classification methodologies Saturation Analysis (SAN) and Saturation Peak Analysis (SPA) <cit.>. SAN and SPA codes are provided by <cit.> []. SAN and SPA take advantage of the saturation channel of the HSV color space. In <cit.>, the authors show that the histogram bin curve of saturation channel is strongly correlated with the presence of smoke, as smoke images contain more low-saturation pixels and that is helpful for classifying smoke/non-smoke images. In their method, a threshold t_c is set empirically. For SAN method, smoke image is recognized if majority of the bin values are below t_c. For SPA method, smoke image is classified by the number of peaks found below and above t_c. We use the default parameters suggested by Leibetseder et al. <cit.> for both SAN and SPA where t_c is set to 0.35.

It is evident from Table <ref> that FC_AVG shows improved performance over the saturation histogram based classification methodologies: SAN and SPA. Precisely, FC_AVG is 1/5% and 1/6% better than the SAN and SPA methods in accuracy/F1-Score metrics. Further in Figure <ref> (b), we show the ROC curve for FC_AVG, SAN and SPA classification methodologies. 









§ CONCLUSION
 
We present a method to enhance RGB images using weighted least squares filter. Our method successfully enhances the informative features in the images for discrimination of smoke/non-smoke images. We demonstrate our proposed method on Cholec80 dataset. Our proposed method obtains significant gains over the baseline RGB images and other enhancement methods. Our method also outperforms the saturation histogram based classification methodologies. Our enhancement method can be used to perform explicit enhancement of image texture and structure features for other tasks too which in turn can lead to more improved performance.

In future work, we plan to further investigate our method on the complete Cholec80 dataset, and also train a CNN architecture with our enhanced RGB images.















IEEEtran


