title: Lymph Node Gross Tumor Volume Detection and Segmentation via Distance-based Gating using 3D CT/PET Imaging in Radiotherapy
authors: Zhuotun Zhu Dakai Jin Ke Yan Tsung-Ying Ho Xianghua Ye Dazhou Guo Chun-Hung Chao Jing Xiao Alan Yuille Le Lu
text: 
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  Z. Zhu et al.
  
  
  PAII Inc., Bethesda MD, USA Johns Hopkins University, Baltimore MD, USA
  Chang Gung Memorial Hospital, Linkou, Taiwan, ROC National Tsing Hua University, Hsinchu City, Taiwan, ROCThe First Affiliated Hospital Zhejiang University, Hangzhou, China
  Ping An Technology, Shenzhen, China
  
  
  GTVLN Detection and Segmentation via Distance-based Gating
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  Lymph Node Gross Tumor Volume Detection and Segmentation via Distance-based Gating using 3D CT/PET Imaging in Radiotherapy
      Zhuotun Zhu1,2Dakai Jin1Ke Yan1Tsung-Ying Ho3Xianghua Ye5Dazhou Guo1Chun-Hung Chao4Jing Xiao6Alan Yuille2Le Lu1
      December 30, 2023
  ==========================================================================================================================
  
  
  
  
  Finding, identifying and segmenting suspicious cancer metastasized lymph nodes from 3D multi-modality imaging is a clinical task of paramount importance. In radiotherapy, they are referred to as Lymph Node Gross Tumor Volume (GTV_LN). Determining and delineating the spread of GTV_LN is essential in defining the corresponding resection and irradiating regions for the downstream workflows of surgical resection and radiotherapy of various cancers. In this work, we propose an effective distance-based gating approach to simulate and simplify the high-level reasoning protocols conducted by radiation oncologists, in a divide-and-conquer manner. GTV_LN is divided into two subgroups of “tumor-proximal" and “tumor-distal", respectively, by means of binary or soft distance gating. This is motivated by the observation that each category can have distinct though overlapping distributions of appearance, size and other LN characteristics. A novel multi-branch detection-by-segmentation network is trained with each branch specializing on learning one GTV_LN category features, and outputs from multi-branch are fused in inference. The proposed method is evaluated on an in-house dataset of 141 esophageal cancer patients with both PET and CT imaging modalities. Our results validate significant improvements on the mean recall from 72.5% to 78.2%, as compared to previous state-of-the-art work. The highest achieved GTV_LN recall of 82.5% at 20% precision is clinically relevant and valuable since human observers tend to have low sensitivity (∼80% for the most experienced radiation oncologists, as reported by literature <cit.>).
  
  
  
  
  
  
  
  § INTRODUCTION
  
  
  Assessing the lymph node (LN) status in oncology clinical workflows is an indispensable step for the precision cancer diagnosis and treatment planning, e.g., radiation therapy or surgical resection. The class of enlarged LN is defined by the revised RECIST guideline <cit.> if its short axial axis is more than 10-15 mm in computed tomography (CT). In radiotherapy treatment, both the primary tumor and all metastasis suspicious LNs must be sufficiently treated within the clinical target volume with the proper doses <cit.>. We refer these LNs as lymph node gross tumor volume or GTV_LN, which includes enlarged LNs, as well as smaller ones that are associated with a high positron emission tomography (PET) signal or any metastasis signs in CT <cit.>. Accurately identifying and  delineating GTV_LN, to be spatially included in the treatment area, is essential for a desirable cancer treatment outcome <cit.>.
  
  It is an extremely challenging and time-consuming task to identify GTV_LN, even for experienced radiation oncologists. High-level sophisticated clinical reasoning guidelines are needed, leading to the risk of uncertainty and subjectivity with high inter-observer variabilities <cit.>. It is arguably more difficult than detecting the more general enlarged LNs. (1) Finding GTV_LN is often performed using radiotherapy CT (RTCT) that (unlike diagnostic CT) is not contrast-enhanced. Hence the metastasis signs for identifying GTV_LN are subtler. (2) GTV_LN itself has poor contrast. Because of the shape and appearance ambiguity, it can be easily confused with vessels or muscles. (3) The size and shape of GTV_LN vary considerably with large amounts of smaller ones that are harder to detect.  Refer Fig. <ref> (top row) for an illustration of GTV_LN. While many previous works attempt to detect enlarged LNs using contrast-enhanced CT  <cit.>, no work, as of yet, has studied the GTV_LN detection in non-contrast RTCT scans. Given the evident differences between the enlarged LNs and GTV_LN, further innovations are required for the robust GTV_LN detection and segmentation. 
  
  
  Valuable insights from physicians' clinical diagnosis and analysis process can be leveraged to tackle this problem. As one of the primary cues, human observers condition the analysis of GTV_LN based on the LNs' distance with respect to the corresponding primary tumor location. For LNs proximal to the tumor, physicians more readily identify them as GTV_LN in radiotherapy treatment. However, for LNs distal to the tumor, they use more strict criteria to include if there are clear signs of metastasis,  e.g., enlarged size, increased PET signals, and/or other CT based evidence <cit.>. Hence, the distance measure relative to the primary tumor plays a key role during physician's decision making. Besides the distance, the PET modality is also of high importance. Although as a noisy imaging channel, it has shown to be helpful in increasing the GTV_LN detection sensitivity <cit.>. As demonstrated in Fig. <ref> (bottom row), PET provides critically distinct information, yet, it also exhibits false positives (FPs) and false negatives (FNs).
  
  
  
  In this paper, we imitate the physician's diagnosis process to tackle the problem of GTV_LN detection and segmentation. (1) We introduce a distance-based gating strategy in a multi-task framework to divide the underlying GTV_LN distributions into “tumor-proximal" and “tumor-distal" categories and solve them accordingly. Specifically, a multi-branch network is proposed to adopt a shared encoder and two separate decoders to detect and segment the “tumor-proximal" and “tumor-distal" GTV_LN, respectively. A distance-based gating function is designed to generate the corresponding GTV_LN sample weights for each branch. By applying the gating function at the outputs of decoders, each branch is specialized to learn the “tumor-proximal" or “tumor-distal" GTV_LN features that emulates physician's diagnosis process. (2) We leverage the early fusion (EF) of three modalities as input to our model, i.e., RTCT, PET and 3D tumor distance map (Fig. <ref>(bottom row)). RTCT depicts anatomical structures capturing the intensity, appearance and contextual information, while PET provides metastasis functional activities. Meanwhile, the tumor distance map further encodes the critical distance information in the network. Fusion of these three modalities together can effectively boost the GTV_LN identification performance. (3) We evaluate on a dataset comprising 651 voxel-wise labeled GTV_LN instances in 141 esophageal cancer patients, as the largest GTV_LN dataset to date for chest and abdominal radiotherapy. Our method significantly improves the detection mean recall from 72.5% to 78.2%, compared with the previous state-of-the-art lesion detection method <cit.>. The highest achieved recall of 82.5% is also clinically relevant and valuable. As reported in <cit.>, human observers tend to have relatively low GTV_LN sensitivities, e.g.,  ∼80% by even very experienced radiation oncologists. This demonstrates our work's clinical values. 
  
  
  
  § METHOD
  
  
  Fig. <ref> shows the framework of our proposed multi-branch GTV_LN detection-by-segmentation method. Similar to <cit.> which are designed for the pancreatic tumors, we detect GTV_LN by segmenting them. We first compute the 3D tumor distance transformation map (Sec. <ref>), based on which any GTV_LN is divided into the tumor-proximal or tumor-adjacent subcategory. Next, a multi-branch detection-by-segmentation network is designed where each branch focuses on one subgroup of GTV_LN segmentation (Sec. <ref>). This is achieved by applying a binary or soft distance-gating function imposed on the penalty function at the output of the two branches (Sec. <ref>). Hence, each branch can learn specific parameters to specialize on segmenting and detecting the tumor-proximal and tumor-adjacent GTV_LN, respectively. 
  
  
  
  
  
  
   §.§ 3D Tumor Distance Transformation
  
  To stratify GTV_LN into tumor-proximal and tumor-distal subgroups, we first compute the 3D tumor distance transformation map, denoted as 𝐗^D, from the primary tumor 𝒪. The value at each voxel x_i represents the shortest distance between this voxel and the mask of the primary tumor. Let B(𝒪) be a set that includes the boundary voxels of the tumor. The distance transformation value at a voxel x_i is computed as
  
  
      𝐗^D(x_i) = {[ q∈ B(𝒪)min d(x_i,q)          if   x_i∉𝒪;                   0          if   x_i∈𝒪 ]. ,
  
  where d(x_i,q) is the Euclidean distance from x_i to q. 𝐗^D can be efficiently computed using algorithms such as the one proposed in <cit.>. Based on 𝐗^D, GTV_LN can be divided into tumor-proximal and tumor-distal subgroups using either binary or soft distance-gating function as explained in detail in Sec. <ref>.
  
  
  
   §.§ Multi-branch Detection-by-Segmentation via Distance Gating
   
  GTV_LN identification is implicitly associated with their distance distributions to the primary tumor in the diagnosis process of physicians. Hence, we divide GTV_LN into tumor-proximal and tumor-distal subgroups and conduct detection accordingly. To do this, we design a multi-branch detection-by-segmentation network with each branch focusing on segmenting one GTV_LN subgroup. Each branch is implemented by an independent decoder to learn and extract the subgroup specific information, while they share a single encoder to extract the common GTV_LN image features. Assuming there are N data samples, we denote a dataset as 𝐒={(𝐗^CT_n, 𝐗^PET_n, 𝐗^D_n,  𝐘_n)}_n=1^N, where 𝐗^CT_n, 𝐗^PET_n, 𝐗^D_n and 𝐘_n represent the non-contrast RTCT, registered PET, tumor distance transformation map, and ground truth GTV_LN segmentation mask, respectively. Without the loss of generality, we drop n for conciseness in the rest of this paper. The total number of branches is denoted as M, where M=2 in our case. A CNN segmentation model is denoted as a mapping function 𝔼: 𝐏 = 𝐟(𝒳; Θ), where 𝒳 is a set of inputs, which consists of a single modality or a concatenation of multiple modalities. Θ indicates model parameters, and 𝐏 means the predicted probability volume.  Given that p(y_i | x_i; Θ_m) represents the predicted probability of a voxel x_i ∈𝒳 being the labeled class from the mth branch, the overall negative log-likelihood loss aggregated across M branches can be formulated as:
  
  
      ℒ = ∑_mℒ_m(𝒳; Θ_m, G_m) = -∑_i∑_mg_m,ilog(p(y_i | x_i; Θ_m)),
  
  where G={G_m}_m=1^M is introduced as a set of volumes containing the transformed gating weights at each voxel based on its distance to the primary tumor. At every voxel x_i ∈G, the gating weights satisfies ∑_mg_m,i = 1. 
  
  
  
   §.§ Distance-based Gating Module
  
  Based on the tumor distance map 𝐗^D, our gating functions can be designed to generate appropriate GTV_LN sample weights for different branches so that each branch specializes on learning the subgroup specific features. In our case, we explore two options: (1) binary distance gating and (2) soft distance gating. 
  
  Binary Distance Gating (BG). Based on the tumor distance map 𝐗^D, we divide image voxels into two groups, x_prox and x_dis, to be tumor-proximal and tumor-distal, respectively, where prox={i|x_i^D≤ d_0, x_i^D∈𝐗^D} and dis={i|x_i^D>d_0, x_i^D∈𝐗^D}. Therefore the gating transformations for two decoders are defined as 𝐆_𝐩𝐫𝐨𝐱=1[x_i^D≤ d_0] and 𝐆_𝐝𝐢𝐬𝐭=1-𝐆_𝐩𝐫𝐨𝐱, where 1[·] is an indicator function which equals one if its argument is true and zero otherwise. In this way, we divide the GTV_LN strictly into two disjoint categories, and each branch focuses on decoding and learning from one category. 
  
  Soft Distance Gating (SG). We further explore a soft gating method that linearly changes the penalty weights of GTV_LN samples as they are closer or further to the tumor. This can avoid a sudden change of weight values when samples are near the proximal and distal category boundaries. Recommended by our physician, we formulate a soft gating module based on 𝐗^D as following:
  
      𝐆_𝐩𝐫𝐨𝐱(x_i) = {[ 1 - x_i^D-d_prox/d_dist - d_prox       if   d_prox< x_i^D≤ d_dist;                                1               if   x_i^D≤ d_prox;                                0               if   x_i^D> d_dist ]. ,
  
  and 𝐆_𝐝𝐢𝐬𝐭(x_i)=1-𝐆_𝐩𝐫𝐨𝐱(x_i) accordingly. 
  
  
  
  § EXPERIMENTAL RESULTS
  
  
  
   §.§ Dataset and Preprocessing
  
  Dataset. We collected 141 non-contrast RTCTs of esophageal cancer patients, with all undergoing radiotherapy treatments. Radiation oncologists labeled 3D segmentation masks of the primary tumor and all GTV_LN. For each patient, we have a non-contrast RTCT and a pair of PET/CT scans. There is a total of 651 GTV_LN with voxel-wise annotations in the mediastinum or upper abdomen regions, as the largest annotated GTV_LN dataset to-date. We randomly split patients into 60%, 10%, 30% for training, validation and testing, respectively. 
  
  Implementation Details.
  In our experiments, PET scan is registered to RTCT using the similar method described in <cit.>. Then all coupling pairs of RTCT and registered PET images are resampled to have a consistent spatial resolution of 1×1×2.5 mm. To generate the 3D training samples, we crop sub-volumes of 96×96×64 from the RTCT, registered PET and the tumor distance map around each GTV_LN as well as randomly from the background. For the distance-gating related parameters, we set d_0=7 cm as the binary gating threshold, and d_prox=5 cm and d_dist=9 cm as the soft gating thresholds, respectively, as suggested by our clinical collaborator. We further apply random rotations in the x-y plane within 10 degrees to augment the training data.
  
  Detection-by-segmentation models are trained on two NVIDIA Quadra RTX 6000 GPUs with a batch size of 8 for 50 epochs. The RAdam <cit.> optimizer with a learning rate of 0.0001 is used with a momentum of 0.9 and a weight decay of 0.0005. For inference, 3D sliding windows with a sub-volume of 96×96×64 and a stride of 64×64×32 voxels are processed. For each sub-volume, predictions from two decoders are weighted and aggregated according to the gating transformation 𝐆_𝐦 to obtain the final GTV_LN segmentation results.
  
  Evaluation Metrics. 
  We first describe the hit criteria, i.e., the correct detection, for our detection-by-segmentation method. For an GTV_LN prediction, if it overlaps with any ground-truth GTV_LN, we treat it as a hit provided that its estimated radius is similar to the radius of the ground-truth GTV_LN within the range of [0.5, 1.5]. The performance is assessed using the mean and max recall (mRecall and Recall_max) at a precision range of [0.10, 0.50] with 0.05 interval, and the mean free response operating characteristic (FROC) at 3,4,6,8 FPs per patient. These operating points were chosen after confirming with our physician. 
  
  Comparison Setups.
  Using the binary and soft distance-based gating function, our multi-branch GTV_LN detection-by-segmentation method is denoted as multi-branch BG and multi-branch SG, respectively. We compare against the following setups: (1) a single 3D UNet <cit.> trained using RTCT alone or the early fusion (EF) of multi-modalities (denoted as single-net method); (2) Two separate UNets trained with the corresponding tumor-proximal and tumor-distal GTV_LN samples and results spatially fused together (our preliminary work <cit.> denoted as multi-net BG); and (3) MULAN <cit.>, a state-of-the-art (SOTA) general lesion detection method on DeepLesion <cit.> that contains more than 10,000 enlarged LNs. 
  
  
  
   §.§ Quantitative Results & Discussion
  
  
  Our quantitative results and comparisons are given in Table. <ref>. Several observations can be drawn on addressing the effectiveness of our proposed methods. (1) The multi-modality input, i.e., early fusion (EF) of RTCT, PET and tumor distance map, are of great benefits for detecting the GTV_LN. There are drastic performance improvements of absolute 6.7% and 7.2% in mRecall and mFROC when EF is adopted as compared to using RTCT alone. These results validate that input channels of PET functional imaging and 3D tumor distance transform map are valuable for identifying GTV_LN. (2) The distance-based gating strategies are evidently effective as the options of multi-net BG, multi-branch BG and multi-branch SG consistently increase the performance. For example, the multi-net BG model achieves 74.7% mRecall and 69.5% mFROC, which is a 1.6% and 1.9% improvement against the best single-net model (where no distance-based stratification is used). The performance further boosts with the network models of multi-branch BG and multi-branch SG, to the highest scores of 78.2% mRecall and 72.4% mFROC achieved by the multi-branch SG. 
  
  
  
  Multi-branch versus Multi-net. Using the distance-based gating strategy, our proposed multi-branch methods perform considerably better than the multi-net BG model. Even our second best model multi-branch BG, the mean and maximal recalls have been improved by 1.4% (from 74.7% to 76.1%) and 2.0% (from 82.5% to 84.5%) against the multi-net BG model. When the multi-branch framework is equipped with the soft-gating, marked improvements of absolute 3.5% and 2.9% in both mRecall and mFROC are observed as compared against to the multi-net BG model. This validates the effectiveness of our jointly trained multi-branch framework design, and our intuition that gradually changing GTV_LN weights for the proximal and distal branches are more natural and effective. As we recall, the multi-net baseline directly trains two separate 3D UNets <cit.> targeted to segment each GTV_LN subgroup. Considering the limited GTV_LN training data (a few hundreds of patients), it can be overfitting prone from the split to even smaller patient subgroups. 
  
  Table. <ref> also compares with the SOTA universal lesion detection method, i.e., MULAN <cit.> on DeepLesion <cit.>. We have retrained the MULAN models using both CT and EF inputs, but even the best results, i.e., using EF, have a large gap (72.5% vs. 78.2% mRecall) with our distance-gating networks, which further proves that the tumor distance transformation cue plays a key role in GTV_LN identification. 
  
  Fig. <ref> illustrates the visualization results of our method compared to other baselines. For the enlarged GTV_LN (top row), most methods can detect it correctly. However, as the size of GTV_LN becomes smaller and the contrast is poorer, our method can still successfully detect them while others struggled.
  
  
  
  
  
  § CONCLUSION
  
  In this work, we propose an effective distance-based gating approach in a multi-task deep learning framework to segment GTV_LN, emulating the oncologists' high-level diagnosis protocols. GTV_LN is divided into two subgroups of “tumor-proximal" and “tumor-distal", by means of binary or soft distance gating. A novel multi-branch detection-by-segmentation network is trained with each branch specializing on learning one subgroup features. We evaluate our method on a dataset of 141 esophageal cancer patients. Our results demonstrate significant performance improvements on the mean recall from 72.5% to 78.2%, as compared to previous state-of-the-art work. The highest achieved GTV_LN recall of 82.5% at the 20% precision level is clinically relevant and valuable.
  
  splncs04
  
  