Accurate Weakly Supervised Deep Lesion Segmentation on CT Scans: Self-Paced 3D Mask Generation from RECIST
Jinzheng Cai Youbao Tang Le Lu Adam P. Harrison Ke Yan Jing Xiao Lin Yang Ronald M. Summers
cs.CV














=







normalsizesetsize10ptxpt
10pt plus2pt minus5pt@
plus3pt6pt plus3pt
minus3ptlistilistI

setsize12ptxipt



§ STARTSECTION SECTION1@1.0EX PLUS
1EX MINUS .2EX.2EX PLUS .2EX




 §.§ startsection subsection2@.2ex
plus 1ex .2ex plus .2ex
 




       

listI
 0pt
 0pt
 0pt
 pt
  makecaption#1#2
 -5pt
 tempboxa#1 : #2
  tempboxa >tempboxatotempboxa
-0.2cm


=0pt =3pt =3pt
=0pt =0pt































Accurate Weakly Supervised Deep Lesion Segmentation on CT Scans: Self-Paced 3D Mask Generation from RECIST
    Jinzheng Cai^1,2[1] [2]   Youbao Tang^1Indicates equal contribution.   Le Lu^1This work is done during Jinzheng Cai's internship at National Institutes of Health. Le Lu is now with Nvidia Corp (lel@nvidia.com).   Adam P. Harrison^1   Ke Yan^1   
 

Jing Xiao^3   Lin Yang^2   Ronald M. Summers^1

National Institutes of Health, Bethesda, MD, 20892, USA^1 

University of Florida, Gainesville, FL, 32611, USA^2 
 

Ping An Insurance (Group) Company of China, Ltd., Shenzhen, 510852, PRC^3 
 
jimmycai@ufl.edu   {youbao.tang, le.lu, adam.harrison, ke.yan}@nih.gov 

xiaojing661@pingan.com.cn   lin.yang@bme.ufl.edu   rms@nih.gov

    December 30, 2023
=======================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================






Volumetric lesion segmentation via medical imaging is a powerful means to precisely assess multiple time-point lesion/tumor changes. Because manual 3D segmentation is prohibitively time consuming and requires radiological experience, current practices rely on an imprecise surrogate called response evaluation criteria in solid tumors (RECIST). Despite their coarseness, RECIST marks are commonly found in current hospital picture and archiving systems (PACS), meaning they can provide a potentially powerful, yet extraordinarily challenging, source of weak supervision for full 3D segmentation. Toward this end, we introduce a convolutional neural network based weakly supervised self-paced segmentation (WSSS) method to 1) generate the initial lesion segmentation on the axial RECIST-slice; 2) learn the data distribution on RECIST-slices; 3) adapt to segment the whole volume slice by slice to finally obtain a volumetric segmentation. In addition, we explore how super-resolution images (2∼5 times beyond the physical CT imaging), generated from a proposed stacked generative adversarial network, can aid the WSSS performance. We employ the DeepLesion dataset, a comprehensive CT-image lesion dataset of 32,735 PACS-bookmarked findings, which include lesions, tumors, and lymph nodes of varying sizes, categories, body regions and surrounding contexts. These are drawn from 10,594 studies of 4,459 patients. We also validate on a lymph-node dataset, where 3D ground truth masks are available for all images. For the DeepLesion dataset, we report mean Dice coefficients of 93% on RECIST-slices and 76% in 3D lesion volumes. We further validate using a subjective user study, where an experienced radiologist accepted our WSSS-generated lesion segmentation results with a high probability of 92.4%. 






§ INTRODUCTION




Assessing lesion or tumor growth rates of multiple time-point scans of the same patient represents one of the most critical problems in imaging-based precision medicine. To track lesion progression using current clinical protocols, radiologists manually conduct this task using response evaluation criteria in solid tumors (RECIST) <cit.>  in computed tomography (CT) or magnetic resonance imaging (MRI) scans. A radiologist first selects an axial image slice where the lesion has the longest spatial extent, then he or she measures the diameters of the in-plane longest axis and the orthogonal short axis. Fig. <ref> provides some visual examples of RECIST marks provided in DeepLesion dataset <cit.>. Typically, the selected RECIST-slice represents the lesion at its maximum cross area. RECIST is often subjective and prone to inconsistency among different observers, especially when selecting the corresponding slices at different time-points where diameters are measured. However, consistency is critical in assessing actual lesion growth rates, which directly impact patient treatment options. On the other hand, if the volumetric lesion measurements between baseline and follow-ups can be accurately computed and compared, this would avoid the subjective selection of RECIST-slices and allow more holistic and accurate quantitative assessment of lesion growth rates. Unfortunately, full volumetric lesion measurements are too labor intensive to obtain. For this reason, RECIST is treated as the default, but imperfect, clinical surrogate of measuring lesion progression. 

Since the clinical adoption of the RECIST criteria, many modern hospitals' picture archiving and communication systems (PACS) and radiology information systems (RIS) have stored tremendous amounts of lesion diameter measurements linked to lesion CT and MRI images. In this paper, we tackle the challenging problem of leveraging existing RECIST diameters to produce fully volumetric lesion segmentation in 3D. Our approach is a weakly supervised semantic segmentation setup, using convolutional neural networks (CNNs) as the core building block. The overall method flow-chart is given in Fig. <ref>. DeepLesion database <cit.> that we exploit is composed of a very large number of 32,735 significant clinical radiology findings (lesions, tumors, lymph nodes, etc) from 10,594 studies of 4,459 patients, bookmarked and measured via RECIST diameters by physicians as part of their day-to-day work. The lesion instances are also clustered into eight different anatomical categories as shown in Fig. <ref>. Without loss of generality, here we only consider CT lesion images. 

From any input CT image with RECIST-measured diameters, we aim to segment the lesion region on the RECIST-selected image first in a weakly supervised manner, followed by generalizing the process into other successive slices to finally obtain the lesion's full volume segmentation. More specifically, with the bookmarked long and short RECIST diameters, we initialize the segmentation using unsupervised learning methods (, GrabCut <cit.>). Afterwards we employ an iterative segmentation refinement via a supervised deep neural network model <cit.>, which can segment the lesion with good accuracy on the RECIST-slice. Importantly, the resulting CNN lesion segmentation model, trained from all training instances of CT RECIST-slices, can capture the lesion image appearance distribution. Thus, the model is  capable of detecting lesion regions from images off the corresponding RECIST-slice. With more slices segmented, more image data can be extracted and used to fine-tune the deep CNN lesion segmentation model. As such, the proposed weakly supervised segmentation model is a self-paced label-map propagation process, from the RECIST-slice to the whole lesion volume. Therefore, we leverage a large amount of retrospective (yet clinically annotated) imaging data to automatically achieve the final 3D lesion volume measurement and segmentation. 

To further enhance our WSSS model, we also investigate the role of image enhancement. A large portion of clinically significant findings in DeepLesion are spatially small (5∼10 mm), which often require significant zooming by radiologists to make precise diameter measurements. To emulate this radiological practice, we develop a stacked generative adversarial network (SGAN) <cit.> model to perform resolution magnification, noise reduction, contrast adjustment and boundary enhancement from the original resolution CT images. We stack two GANs together, where the first reduces the noise from original CT image and the second generates a higher resolution image with enhanced boundaries and high contrast. Because the CT modality can only be imaged at a limited range spatial resolution (∼ 1 mm/pixel) and super-resolution CT images (, ∼0.2 mm/pixel) do not physically exist, we train our SGAN on a large quantity of paired low- and high-quality natural images and use transfer learning to apply the model to CT images.

We evaluate the proposed WSSS segmentation method using all lesion categories <cit.>. To the best of our knowledge, this is the first work to develop a class-agnostic lesion segmentation approach. For quantitative evaluation, we manually annotated 1,000 RECIST-slices and 200 lesion volumes, where the proposed WSSS achieves mean Dice coefficients of 93% and 76%, respectively. To compare WSSS against a fully-supervised approach, we also validate on a lymph node (LN) dataset <cit.>, consisting of 984 LNs with full pixel-wise annotations. Finally, we also conduct a subjective user study, and demonstrate that radiologist accept the WSSS generated lesion masks with high probability of 92.4%. 
In summary, we present a weakly supervised semantic segmentation approach for accurate lesion segmentation and volume measurements in the wild, using a comprehensive dataset <cit.>. With no extra human annotation efforts required, we convert tens of thousands of recorded RECIST 2D diameter based measurements in PACS/RIS into accurate 3D lesion volume segmentation assessments. 








§ RELATED WORK
 
Weakly supervised semantic segmentation is a challenging computer vision task that has drawn considerable interest. Recently promising results have been reported <cit.>. Most weakly supervised semantic segmentation methods update pixel labels and CNN models using training iterations. The pixel-level training label is often initialized from weak annotations, , bounding boxes of foreground objects. Subsequently, CNNs are trained to capture the initial masks. Although many pixels are possibly initially mislabelled, the CNNs are expected to robustly handle label noise (to some extent) and model the complete data distribution. Once the CNN model converges, its output is used to obtain more accurate pixel labels. Updates between CNN and pixel training labels loop iteratively until no further changes can be measured. However, as observed in <cit.>, very high-quality initial pixel labels can allow the CNN model to converge in as few as 1∼2 iterations.

Thus, it is essential to have the object mask “correctly” initialized and updated. A popular mask initialization uses multiscale combinatorial grouping (MCG) <cit.> to group superpixels into plausible object parts, using a bounding box to delineate foreground, background, and uncertain regions during CNN training <cit.>. RECIST diameters are similar to scribbles over foreground objects, and these can be used to coarsely delineate superpixels into foreground and background <cit.>. However, empirically MCG does not perform well under the CT imaging modality (compared to natural images) and thus is not suitable for our lesion segmentation task. Semantic edge based methods, like GrabCut^+ <cit.>, are also not applicable because unsupervised edge detection methods, such as gPb <cit.>, suffer from abundant false positives or perform unstably across the whole CT image dataset. 

Another stream of related work is extending weakly supervised semantic segmentation from static images to video <cit.>. In videos, the same object often retains similar appearance across different temporal frames even though the locations may vary. Using this prior, the training images of object-wise labels can be extracted and propagated across video frames to facilitate CNN training. 
However lesions exhibit not only positional movements but also drastic morphological changes across CT slices, adding further challenges to lesion volume segmentation from RECIST diameters. Hence, our work differs from prior art by employing a principled trimap mask initialization and GrabCut <cit.>, coupled with self-paced learning <cit.> to progressively refine the 3D lesion segmentations. 

Recently, deep CNN based image super-resolution methods <cit.> have achieved state-of-the-art performance, due to CNN's capability to learn powerful features and model long-range contextual information from a collection of low resolution images. Mappings between low and high resolution images are learned by minimizing the mean squared error (MSE) loss or perceptual loss <cit.>. Adversarial learning strategies <cit.> are also employed to train CNN models for better reconstruction of fine details and edges. In this work, we adopt the perceptual loss and adversarial learning to simulate high-resolution images (∼0.2 mm per-pixel) via CNNs directly from the original CT images (∼1.0 mm per-pixel). Unlike other works, we use the enhanced images to improve the WSSS performance.






§ METHOD
 




In DeepLesion dataset <cit.>, manually selected axial CT slices contain RECIST diameters that represent the longest axis and its perpendicular counterpart. Given RECIST diameters as weak supervision, we leverage weakly supervised principles to learn a deep lesion CNN segmentation model using both 2D slice and 3D volumes with no extra pixel-wise manual annotations. Our pipeline for WSSS contains four main steps as follows. 



 §.§ From RECIST to 2D Training Masks
 
We denote elements in the DeepLesion dataset <cit.> as 𝒟={(V^i,R^i)} for i ∈{1,…,N}, where N is the number of lesions, V^i is the CT region of interest, and R^i is the corresponding RECIST diameters. To create the 2D training data for the WSSS model, the image and label map pairs, X^i and Y^i, respectively, must be generated. Together they comprise the set ℱ={(X^i, Y^i)}, for i ∈{1,…,N}. For notational clarity, we drop the superscript i. X=V_r is simply the RECIST-slice, , the axial slice that contains R, and r is its slice number. Both GrabCut <cit.> and densely-connected conditional random fields (DCRF) <cit.> can be adopted to produce the training mask Y. When GrabCut starts with a good quality initial trimap (explained below), we observe that it produces better results, which is consistent with the findings in <cit.>.

More specifically, GrabCut is initialized with the image foreground and background regions and produces a segmentation using iterative energy minimization. Good initializations located near the global energy minimum may aid GrabCut's convergence. We use the spatial prior information, provided by R, to compute a high quality initial trimap T from R, , regions of probable background (PBG), probable foreground (PFG), background (BG), and foreground (FG). Note that unlike the original trimap definition <cit.>, we define four region types. More specifically, if the lesion bounding box tightly around the RECIST axes is [w,h], a [2w,2h] ROI is cropped from the RECIST-slice. The trimap assigns the outer 50% as BG and allocates 10% of the image region, dilated from R as FG. The remaining 40% is divided in half between PFG and PBG based on the distances to FG and BG. Finally, the long/short axes of R roughly clamp the size of GrabCut segmentation, ensuring that the segmentation will not shrink into a small region when the CT image intensity distribution is homogeneous and no clear lesion boundary is presented. Fig. <ref> visually depicts the mask generation process.




 §.§ CNN Lesion Appearance Model

Many image classification or segmentation CNN models have been presented in recent works <cit.>. Without loss of generality, we use the holistically-nested network (HNN) <cit.> and UNet <cit.> as our baselines, both of which provide state-of-the-art, yet straightforward semantic architectures. We represent our CNN models as a mapping function Ŷ = f(X;θ), with the goal of optimizing θ to minimize the differences between the current model outputs Ŷ and ground truth training labels Y. Although the GrabCut masks are imperfect, we use them as Y for the next step of CNN model training, expecting that the CNN will generalize well even with considerable label noise. Thus, we aim to minimize

    L(ℱ; θ) = 1/N∑_i=1^N∑_m∈ Y^i H (ŷ^(i,m),y^(i,m))/|Y^i|,

where m is the pixel index inside Y, |Y| is the cardinality of Y, and H(·) is the cross-entropy function. 




 §.§ Self-Pacing for Volume Segmentation
 
For obtaining 3D volume segmentations, we follow a similar strategy as with the RECIST slices, except in the 3D case, we must infer R for off-RECIST slices and also incorporate inferences from the CNN model. These two priors are used together for self-paced CNN training.

3D RECIST Estimation: A simple way to generate off-RECIST-slice  R diameters is to take advantage of the fact that RECIST-slice R lies on the maximal cross-sectional area of the lesion. The rate of reduction of off-RECIST-slice endpoints is then calculated by their relative distance to the intersection point of the major and minor axes. Estimated 3D RECIST endpoints are then projected from the actual RECIST endpoints by Pythagorean theorem using physical Euclidean distance.

Trimap from CNN Output: Different from Sec. <ref>, trimap generation now takes both the CNN output Ŷ and the estimated RECIST R̂ as inputs: T̂ = trimap(Ŷ,R̂,R). The Ŷ is first binarized by adjusting the threshold so that it covers at least 50% of R's pixels. Regions in Ŷ that associate with high foreground probability values and overlap with R will be set as FG together with R. Similarly, regions with high background probabilities and that have no overlap with R will be assigned as BG. The remaining pixels are left as uncertain using the same distance criteria as in the 2D mask generation case and fed into GrabCut for lesion segmentation. In the limited cases where the CNN fails to detect any foreground regions, we generate T̂ solely from the estimated RECIST R̂. We observe that when possible, GrabCut initialized by the CNN output performs better than being solely initialized from R̂.

Self-Paced CNN Training: To generate 3D lesion segmentations from 2D RECIST annotations, we train the CNN model in a self-paced or self-taught mechanism. It begins by learning the lesion appearances via CNN on RECIST-slices. After the model converges, Y=GrabCut(trimap(Ŷ,R̂,R),X) is used to generate the training masks on successive slices so that it can be further fine-tuned on more newly “harvested” images. We progressively expand the extent of the harvested images, to include more CT slices along the longitudinal axis. Taking one lesion volume for example, the CNN is first trained on its RECIST slice X until convergence, we then apply this CNN model to slices [V_r-1,V_r+1] to compute the predicted probability maps [Ŷ_r-1,Ŷ_r+1]. Given these probability maps, we create trimaps [T̂_r-1,T̂_r+1] to employ GrabCut refinement <cit.> on these slices. The GrabCut method consequently generates the training labels that are used by the CNN model in the next training round on the slices [V_r-2,V_r+2] slices. As this procedure proceeds iteratively, we can gradually obtain the converged lesion segmentation result in 3D. We summarize our WSSS method in Algorithm <ref> and visually depict its process in Fig. <ref>. Note that it is possible that GrabCut(·) gives no or very few foreground areas or pixels if the initial trimap and CT image are of bad quality. In these cases, we set up foreground, background and ignored uncertain pixels by combining the R̂ and the CNN output. CNN predicted foreground regions that overlap with R̂ are assigned as foreground; predicted background regions that have no overlap with RECIST are set as background; with the remaining pixels ignored during training.





 §.§ Stacked-GAN Assisted Segmentation


We next investigate improving lesion segmentation accuracy by enhancing the quality and resolution of the input CT images via GANs. A direct application of super-resolution generative adversarial networks (SRGANs) <cit.> on CT images produces many artificial, noisy edges and even deteriorates the final lesion segmentation outcome. Spurred by this observation, we propose a two-stage stacked GAN (SGAN) process that first reduces image noise and then performs object boundary enhancement. More specifically, each stage is conducted using an independent SRGAN <cit.>. During model inference, the output of the first GAN will be fed as input to the second GAN. For lesion segmentation, we use both the denoised and enhanced outputs from SGAN and the original CT image as three-channel inputs since they may contain complementary information.

Synthesized Training Data: Normally the SGAN or SRGAN models are trained with pairs of low- and high-resolution images. This can be obtained easily in natural images (by down-sampling). However physical CT images are imaged by medical scanners at roughly fixed in-plane resolutions of ∼1 mm per-pixel and CT imaging at ultra-high spatial resolutions does not exist. 
For the sake of SGAN training, we leverage transfer learning using a large-scale synthesized natural image dataset: DIV2K <cit.> where all images are converted into gray scale and down-sampled to produce training pairs. For the training of the denoising GAN, we randomly crop 32×32 sub-images from distinct training images of DIV2K. White Gaussian noise at different intensity variance levels σ_i∈(0,50] are added to the cropped images to construct the paired model inputs. For training the image-enhancement GAN, the input images are cropped as 128×128 patches and we perform the following steps: 1) down-sample the cropped image with scale s∈[1,4], 2) implement Gaussian spatial smoothing with σ_s∈(0,3], 3) execute contrast compression with rates of κ∈[1,3], and 4) conduct up-sampling with the scale s to generate images pairs.

To fine-tune using CT images, we process 28,000 training RECIST slices using the current trained SGAN and select a subset of up to 1,000 slices by subjectively inspecting the CT super-resolution results. The selected CT images are subsequently added to the training for the next round of SGAN fine-tuning. This iterative process finishes when no more visual improvement could be observed.




§ EXPERIMENTS
 

Datasets: The DeepLesion dataset <cit.> is composed of 32,735 bookmarked CT lesion instances (with RECIST measurements) from 10,594 studies of 4,459 patients. Lesions have been categorized into the 8 subtypes of lung, mediastinum, liver, soft-tissue, abdomen, kidney, pelvis, and bone. To facilitate quantitative evaluation, we segmented 1,000 testing lesion RECIST-slices manually. Out of these 1000, 200 lesions (∼3,500 annotated slices) are fully segmented in 3D as well. Additionally, we also employ the lymph node (LN) dataset[<https://wiki.cancerimagingarchive.net/display/Public/CT+Lymph+Nodes>] <cit.>, which consists of 389 mediastinal and 595 abdominal LNs from 176 CT scans with complete pixel-wise annotations. Enlarged LNs are a lesion subtype and producing accurate segmentations is quite challenging even with fully supervised learning <cit.>. Importantly, the fully annotated LN dataset can be used to evaluate our WSSS method against an upper performance limit, by comparing results with a fully supervised approach <cit.>. 

Pre-processing: For the LN dataset, annotation masks are converted into RECIST diameters by measuring the major/minor axes after localizing the 5 successive axial slices nearest to the largest LN cross-section. For robustness, up to 20% random noise is injected in the RECIST diameter lengths to mimic the uncertainty of manual annotation by radiologists. For both datasets, based on the location of RECIST bookmarks, CT ROIs are cropped at two times the extent of the lesion's longest diameters so that sufficient visual context is preserved. The dynamic range of each lesion ROI is then intensity-windowed properly using the CT windowing meta-information in <cit.>. The LN dataset is separated at the patient level, using the split of 80% and 20% for training and testing, respectively. For the DeepLesion dataset <cit.>, we use 28,000 lesion volumes for training and the rest for testing.

Evaluation: The mean dice similarity coefficient (mDICE), DICE=2TP/(2TP+FP+FN) and the pixel-wise precision and recall scores are used to evaluate the quantitative segmentation accuracy. We measure volumetric similarity (VS) VS=1-(FN-FP)/(2TP+FP+FN) and averaged Hausdorff distance (AVD) to better capture and evaluate the small but vital changes near object boundaries <cit.>.



Our baseline CNN model is the holistically-nested network (HNN), originally proposed for natural image edge detection <cit.>, which has been adapted successfully for lymph node <cit.>, pancreas <cit.>, and lung segmentation <cit.>. In all experiments, CNN training is implemented in Tensorflow <cit.> and Tensorpack [<https://github.com/ppwwyyxx/tensorpack>] with Adam Optimizer and initialized from the pre-trained ImageNet model <cit.>. The learning rate is set as 5×10^-5 and it drops to 1×10^-5 when the model training-validation plot plateaus.



































Label Map Initializations: Both GrabCut <cit.> and densely-connected conditional random fields (DCRF) <cit.> are extensively evaluated for initializing training label maps from RECIST diameters. We test two alternatives to the trimap approach explained in Sec. <ref>, which we denote GrabCut-R. Both alternatives are based on a tight bounding box (bbox), which matches the extent of the lesion RECIST marks with 25% padding (against the lesion's spatial extent). The first alternative (GrabCut), sets the area outside bbox as background and area inside bbox as probable foreground. The second alternative (GrabCut^i) sets the central 20% bbox region as foreground, regions outside the bbox as background, and the rest as uncertain. This is similar to the setting of bbox^i in <cit.>. We also test DCRF, using bbox^i as the unary potentials and intensities to compute pairwise potentials <cit.>. We empirically found that DCRF is moderately sensitive to parameter variations. The optimal DCRF performance is reported in Table <ref>. Finally, we also report results when we directly use the RECIST diameters, but dilated to 20% of bbox area, which unsurprisingly produces the best precision, but at the cost of very low recall. However, as can be seen in the table, GrabCut-R significantly outperforms all alternatives, demonstrating the validity of our mask initialization process. 






































CNN Training under Different Levels of Supervision: Following Sec. <ref>, there are three ways of generating initial lesion masks on the RECIST-slice: using the trimap from the dilated RECIST (ignoring uncertain regions in training), GrabCut-R (processed from the trimap) and the full pixel-wise manual annotation (when available). Using the LN dataset <cit.> with manual ground truth, HNNs trained from  these three initializations of label maps achieve 61%, 70%, and 71% mDICE scores, with increased levels of supervision respectively. No extra segmentation refinement options like adaptive sample mining, or weighted training loss are applied. This observation demonstrates the robustness and effectiveness of using GrabCut-R label map initialization, which only performs slightly worse than the fully annotated pixel-wise masks. On the DeepLesion <cit.> testset of 1,000 annotated testing RECIST-slices, HNN trained on GrabCut-R initialization outperforms the deep model learned from dilated RECIST maps by a margin of 25% in mDICE (90.6% versus 64.4%). GrabCut post-processing further improves the results from 90.6% to 91.5%. We also aim to demonstrate that our WSSS approach, trained on a large quantity of weakly supervised or “imperfectly-labeled” object masks, can outperform fully-supervised models trained on less data. To do this, we separated the 1,000 annotated testing images into five folds and report the mDICE scores over 5-fold cross-validation using fully-supervised HNN <cit.> and UNet <cit.> models. Impressively, the 90.6% dice score of WSSS considerably outperforms the fully supervised HNN and UNet mDICE scores of 84% and 73%, respectively. This demonstrates the importance of training CNNs from a large-scale “imperfectly-labeled” dataset. All the results are described in Table <ref>. 




















3D Segmentation: In Fig. <ref>, we show the segmentation results on 2D CT slices arranged in the order of voxel-distance with respect to the RECIST-selected slice. GrabCut with 3D RECIST estimation (GrabCut-3DE) produces good segmentations (∼91%) on the RECIST-slice, but can degrade to 55% mDICE when the off-slice distance raises to 4. This is mainly because 3D RECIST approximation often is not a robust and accurate estimation across slices. 
In contrast, the HNN trained using RECIST-slices generalizes well with large slice offsets. It still gains >70% of mDICE even when the offset distance range as high as 6. However, performance is further improved at higher slice offsets when using self-paced learning with 3 axial slices, , WSSS (mid), and even further when using the full self-paced learning with 5 axial slices, , WSSS.  These results demonstrate the value of using our self-paced learning approach to generalize beyond 2D RECIST slices into full 3D segmentations. Fig. <ref> further demonstrates the model improvements from self-paced learning using precision-recall curves. Unsurprisingly, the WSSS schemes do not provide much improvement on the 2D RECIST slices; however, significant improvements are garnered when considering the full 3D segmentations, again demonstrating the benefits of WSSS to achieve clinically useful volumetric lesion segmentations. The final 3D segmentation results are tabulated in Table <ref>.










Ablation Study of Stacked-GAN: When we train the HNN using two input channels of original CT and super resolution (SR) images, there are significant improvements in the precision, AVD, VS, and mDICE metrics (see Table.<ref>). In particular, the AVD scores drop by 15% from 0.230 to 0.189, likely implying better lesion boundary delineation. Category-wise 2D lesion segmentation results are reported in Table <ref>. The SGAN assisted HNN (HNN-SR) improves over standard HNN, and, with GrabCut post-processing, the best result of 93% mDICE score is achieved.



 §.§ Subjective Clinical Acceptance Evaluation

The main motivation and goal of this work is exploring the feasibility of replacing and converting the current 2D RECIST diameter based lesion measurements into 3D volumetric scores. Consequently, it is critical to evaluate clinician's subjective feedback on the acceptance rate of our WSSS produced lesion segmentations. We measure the acceptance rate of a US board-certificated radiologist with over 25 years of clinical practice experience under two different scenarios.

1) The 1,000 testing RECIST-measured lesion CT images (overlaid with manual or automatic segmentation masks) are randomly displayed to the experienced radiologist to judge whether to accept the segmentation or not. The acceptance rate for WSSS segmentations is 92.4%, which is close to the 98.7% acceptance rate of the manually segmented masks (which were performed by 2 trainees under the supervision of a radiologist). Examples of accepted and rejected automatic segmentations are displayed in Fig. <ref>.

2) We show both manual and WSSS computed lesion segmentation results simultaneously in shuffled orders to the radiologist and let him pick the more preferable result (, the human judge does not know which segmentation mask is the manual ground truth). Consequently, the radiologist picks 72 WSSS-over-manual segmentation cases, designates another 645 instances as inseparable in segmentation quality, and marks the remaining 269 as manual-over-WSSS. 



Finally, we also measure how well the WSSS method can track lesion changes over time. Toward this end, 22 pairs of lesion volumes with follow-ups are selected from our test dataset. They are used to evaluate the volume changes at two time points via the means of manual segmentation, RECIST measurements, or the proposed WSSS segmentation. For RECIST, the lesion volume is estimated by ellipsoid volume as V = π (length × width^2) / 6 where length and width represent the long and short axes, respectively. In Fig. <ref>, the volume change plots against the follow-up cases are illustrated. Manual and computerized segmentation results correlate well (with R^2=0.78), whereas the ellipsoid RECIST estimations tend to report smaller-than-actual volume changes (with R^2=0.62).




§ DISCUSSIONS & CONCLUSION
 

We present a simple yet surprisingly effective weakly supervised deep segmentation approach on turning massive amounts of RECIST-based lesion diameter measurements (retrospectively stored in hospitals' digital repositories) to full 3D lesion volume segmentation and measurements. The radiologist's zooming factor when depicting the lesion diameters on CT images is emulated via our proposed stacked GAN models for image denoising and super-resolution. Importantly, our approach does not require pre-existing RECIST measurements on processing new cases. 

Our method is fully automatic and learned from a large quantity of partially-labeled clinical annotations. The lesion segmentation results are validated through both quantitative evaluation (e.g., 93% mean DICE on RECIST-slices and 76% for 3D lesion volume segmentation) and subjective user study. We demonstrate that our self-paced learning improves performance over state-of-the-art CNNs. Moreover, we demonstrate how leveraging the weakly supervised, but large-scale data, allows us to outperform fully-supervised approaches that can only be trained on subsets where full masks are available. Our 3D lesion segmentation also produces more accurate estimations of lesion volume changes than the RECIST criteria. Our work is potentially of high importance for automated and large-scale tumor volume measurement/management in the domain of precision quantitative radiology imaging.

ieee
	




§ SUPPLEMENTARY MATERIAL





Stacked GAN: In <cit.>, generative adversarial networks (GAN) have been successfully used for natural-image super resolution, which produces high-quality images with more low-level visual details and edges from their low-resolution counterparts. For lesion segmentation, if we can improve the visual clarity and contrast of image edges of lesions, the segmentation performance and accuracy may be subsequently improved. As such, our work is partially inspired by <cit.>. As shown in the first row of Fig. <ref>, CT images are often noisy and suffer from low contrast due to radiation dosage limits. Directly applying GAN-based super resolution methods on such images generates undesirable visual artifacts and edges that are harmful for lesion segmentation accuracy. To address this problem, the CT imaging noise needs to be reduced before super resolution or spatial zooming. It is challenging to train a single GAN model which directly outputs high resolution images with high visual quality (, clear object-level boundaries from noisy images) from the original lesion CT images. Therefore our proposed stacked GAN operates in two stages, breaking the CT-image super resolution process into two sub-tasks: denoising followed by spatial zooming with enhancement.

Given a CT lesion image X_0 (as shown in the first row of Fig. <ref>), we first generate a denoised version of the input image X_1 by employing our first GAN model (consisting of a generator G_1 and a discriminator D_1) that focuses on removing the random image noise. X_1 has the same size as X_0. Although the noise has been reduced in the generated image X_1 (as demonstrated in the second row of Fig. <ref>), lesions have blurry edges and the imaging contrast between lesion and background regions are generally low. However, clear edges and high contrast are desirable and important for achieving high precision lesion segmentation results. As well, a considerable amount of lesions are quite small in size (<10mm or less than 10 pixels according to their long axis diameters) and human observers typically apply zooming (via commercial clinical PACS workstations). If we intend to develop an effective CNN model to learn discriminative imaging features for lesion segmentation, image resolutions should be sufficiently higher than the physical CT imaging resolution (approximately in the range of 1mm per pixel). To solve this issue, our second GAN model, which also contains a generator G_2 and a discriminator D_2, is built upon the output X_1 from the first GAN to produce a high resolution version X_2 (as illustrated in the third row of Fig. <ref>). This high-resolution image provides both clear lesion boundaries and high contrast. Since the three resulting images,  the original, denoised, and high-resolution variants, as a triplet may have complementary information, we compose them together into a three-channel image that is fed into the next lesion segmentation stage.

We adapt similar architectures as <cit.> for the generators and discriminators. In <cit.>, the generator has 16 identical residual blocks and 2 sub-pixel convolutional layers <cit.>, which are used to increase the resolution. Each block contains two convolutional layers with 64 3 × 3 kernels followed by batch-normalization <cit.> and ParametricReLU <cit.> layers. For a trained model, the method <cit.> can only enlarge the input image by fixed amounts. In the DeepLesion dataset, the lesion sizes vary considerably. Different lesions have to be enlarged with correspondingly different zooming factors. Therefore the sub-pixel layers are removed in the high-resolution generator G_2 of our stacked GAN model. Because it is an easier subtask, a simpler architecture that contains just 9 identical residual blocks is designed for the denoising generator G_1 of denoising. G_1 and G_2 are fully convolutional neural networks and can take input images with arbitrary sizes. For the discriminator design, D_1 and D_2, we use the same architecture as <cit.>, which consists of 8 convolutional layers with 3×3 kernels, LeakyReLU activations (α=0.2) and two densely connected layers followed by a final sigmoid layer. The stride settings and numbers of kernels for 8 convolutional layers are (1,2,1,2,1,2,1,2) and (64,64,128,128,256,256,512,512), respectively. 




3D RECIST Estimation: As described in the main text, the estimated RECIST of off-RECIST-slices is projected from the actual RECIST diameters via Pythagorean theorem using physical Euclidean distance. In Fig. <ref>, we visualize the process of RECIST projection, where the length of the long/short axis on the off-RECIST-slice is calculated based on the long/short axis ratio of the actual RECIST diameters. Meanwhile, the intersection of each pair of long and short axes is fixed to the same position as the intersection of the actual RECIST.




Label Map Initializations: We also consider generating the initial training label maps from the images after super-resolution. As shown in Table <ref>, the row of GrabCut-R-SR presents recall,  precision, and mDICE of the training labels produced from applying GrabCut-R on the super-resolution images. However, we observe that the differences between results produced by GrabCut-R or GrabCut-R-SR are not significant. This may be because GrabCut <cit.> cannot fully make use of the augmented image information compared to deep CNN models. Therefore we only use the original CT images to generate training label maps in the main submission. 







3D Segmentation with WSSS: More quantitative evaluations of the proposed weakly supervised self-paced segmentation (WSSS) are presented in Fig. <ref> and Fig. <ref>. These results are based on the HNN based lesion segmentation models that are trained on the RECIST-slice, and under WSSS-3/5/7 settings, which means that the HNN model is trained in a self-paced manner with 3, 5, and 7 axial slices, respectively. Limited improvement is observed from WSSS-7 against WSSS-5 in Fig. <ref>, which indicates the convergence of model training. Therefore, we report the outputs of WSSS-5 as the best 3D segmentation results in the main submission.





Subjective Study: GUI snapshots of our subjective study are displayed in Fig. <ref> and Fig. <ref>. Of note, we allow the radiologist to reject both manual and automatic segmentation in Visual Comparison Study, and this explains why the sum of WSSS-over-manual (72), manual-over-WSSS (269), and inseparable (645) is 986, which is less than 1,000. In our study, 14 pairs of both WSSS and manual segmentation results were rejected at the same time. 



RECIST-Slice Segmentation with Super-resolution CT Images: Finally in Fig. <ref>, we compare the results of RECIST-slice segmentation with and without image super resolution augmentation via our proposed stacked GAN models. We find, in the lesion categories of abdomen and soft-tissue, the super-resolution images preserve sharper lesion boundaries than the original CT images and their corresponding segmentation results are indeed superior in delineating better lesion boundaries with higher accuracy than raw CT images.






