Brain Tumor Synthetic Segmentation in 3D Multimodal MRI Scans
Mohammad Hamghalam Baiying Lei Tianfu Wang

	
	
	
	

	
		

	

	
	Brain Tumor Synthetic Segmentation in 3D  Multimodal MRI Scans
	
	
	
	
	
	M. Hamghalam et al.
	
	
	
	National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, Guangdong Key Laboratory for Biomedical Measurements and Ultrasound Imaging, School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, China, 518060. Faculty of Electrical, Biomedical and Mechatronics Engineering, Qazvin Branch, Islamic Azad
	University, Qazvin, Iran.

	m.hamghalam@gmail.com

	{leiby,tfwang}@szu.edu.cn
	
	
	
	Brain Tumor Synthetic Segmentation in 3D Multimodal MRI Scans
    Mohammad Hamghalam1,20000-0003-2543-0712 Baiying Lei1 Tianfu Wang1
    July 2019
======================================================================


	
	
	The magnetic resonance (MR) analysis of brain tumors is widely used for diagnosis and examination of tumor subregions. The overlapping area among the intensity distribution of healthy, enhancing, non-enhancing, and edema regions makes the automatic segmentation a challenging task. Here, we show that a convolutional neural network trained on high-contrast images can transform the intensity distribution of brain lesions in its internal subregions. Specifically, a generative adversarial network (GAN) is extended to synthesize high-contrast images.
	A comparison of these synthetic images and real images of brain tumor tissue in MR scans showed significant segmentation improvement and decreased the number of real channels for segmentation.
	The synthetic images are used as a substitute for real channels and can bypass real modalities in the multimodal brain tumor segmentation framework. Segmentation results on BraTS 2019 dataset demonstrate that our proposed approach can efficiently segment the tumor areas. In the end, we predict patient survival time based on volumetric features of the tumor subregions as well as the age of each case through several regression models.
		
	
	
	
	
	
	

§ INTRODUCTION
    
	
	
Glioma is the most aggressive and widespread tumor is grouped into low-grade gliomas (LGGs) and high-grade gliomas (HGGs). Multimodal MR channels in BraTS 2019 datasets <cit.>, included of FLAIR, T1, T1c, and T2, are routinely used to segment internal parts of the tumor, i.e., whole tumor (WT), tumor core (TC), and enhancing tumor (ET). Several segmentation approaches have been proposed to segment regions of interest through classic <cit.> and modern machine learning methods, especially brain tumor segmentation techniques <cit.>. 
	



	

	
	
	
			

	
	
The focus of current research is to form a generator that increases the contrast within subregions of the brain tissue. 
The generator, which is a deep neural network model, employes a real channel as input to produce the synthetic one. Our framework comprises two stages: (1) we generate high tissue contrast images based on FLAIR sequence in our convolutional neural network (CNN) model, (2) we train a 3D fully convolutional network (FCN) <cit.> based on the synthetic images to segment region of interests.

	

§ METHOD

	
Our goal is to segment tumor subregions based on multimodal 3D magnetic resonance (MR) volumes. Fig.<ref> demonstrates an overview of the proposed method based on synthetic high-contrast images. In contrast to the current methods, we use both real and synthetic volumes for the segmentation task. Following, we first introduce the synthetic image generator module, based on the generative adversarial networks (GANs) model <cit.>, and then 3D FCN architecture for segmentation is discussed.
	
	
	
	
	

 §.§ Synthetic Image Generator

	
	
We extend the image-to-image translation method <cit.> to deal with the synthesis of high-contrast 2D images. Our model trains on high-contrast images, building based on manual labels, in an adversarial framework. The synthesis model contains a Generator, based on the 2D-U-Net <cit.>, and a Discriminator, build on 2D FCN  network. Fig.  illustrates the image translation framework, where both the generator and the discriminator blocks are trained on FLAIR with a patch size of 128 × 128 pixels. In implementation details, we follow <cit.>, including the number of epochs, the number of layers, and the kernel sizes. For each subject in the BraTS'19 dataset, we provide a 3D synthetic volume for the next stage, segmentation. 
	
	

 §.§ Synthetic Segmentation

	
	The output volumes from synthetic image generator block are concatenated with real modalities (FLAIR, T1c, and T2) and fed into segmentation block to predict region of interests.
	The segmentation network allows jointly capturing features from FLAIR, synthetic, T1c, and T2 modality. For the 3D segmentation block, we rely on ensembling the 3D FCN on axial, sagittal, and coronal planes.
	

	
	

§ EXPERIMENTAL RESULTS

	
	
	

 §.§ Implementation Details

	
	We implement the proposed design employing the KERAS with 12GB NVIDIA TITAN X GPU. We have scaled image patches to sizes 128 × 128 pixels for translation.
	The model is trained through the ADADELTA <cit.> optimizer (learning rate = 0.9, ρ=0.90, epsilon=1e-5). Dropout is employed to avoid over-fitting over the training (p_drop=0.4).

	

 §.§ Datasets

	
	The performance of the proposed method is evaluated on the BraTS'19 dataset, which has two datasets of pre-operative MRI sequences: Training (335 cases) and Validation (125 cases). Each patient is giving 155×240×240 with four channels: T1, T2, T1c, and FLAIR. In the manual label of BraTS'19, there are three tumor regions: non-enhancing tumor, enhancing tumor, and edema. The evaluation is figured out by CBICA IPP[https://ipp.cbica.upenn.edu] online platforms.
	Metrics computed by the online evaluation platforms in BraTS'19 are Dice Similarity Coefficient (DSC) and the 95th percentile of the Hausdorff Distance (HD95). DSC is considered to measure the union of prediction and manual segmentation. It is measured as DSC = 2TP/FP+2TP+FN where TP, FP, and FN are the numbers of true positive, false positive, and false negative detections, respectively. 
	

		
	


 §.§ Segmentation Results on BRATS'19
	

Fig. <ref> shows examples of brain tumor prediction in LGG and HGG slides on BraTS19 along with corresponding labels, where the subject IDs are	"BraTS19-TCIA10-175-1" and "BraTS19-CBICA-APK-1" for LGG and HGG, respectively.	
The results in Table <ref> show that our method performed competitive performance on validation set (125 cases) of BraTS dataset. Results are reported in the online processing platform by BraTS'19 organizer. Moreover, Table <ref> reports the average results on 335 training case of the BraTS'19.  

	

	
	
	

	
		
	
	






§ OVERALL SURVIVAL PREDICTION MODEL


BraTS'19 dataset contains 102 gross total resections (GTR) pre-operative scans out of 335 training cases in which the age of patients is available. These subjects are applied for developing a model to predict the overall survival (OS) of the patient. To this end, we measure the volume of WT, TC, and ET after segmentation to create a feature vector to predict patient OS. We also consider patient's age as an input feature to increase survival prediction accuracy. Thus, we have a 4-dimensional normalized feature vector that scaled between 0 and 1. We train different regression models to predict OS through supervised machine learning, including linear models, regression trees, support vector machines (SVMs) with different kernel functions, Gaussian process regression (GPR) models, and ensembles of trees. We measure root mean square error (RMSE), maximum absolute error (MAE), and prediction speed during inference  (observation/sec) to assess model performance. 
The 5-fold cross-validation is applied to evaluate these models with four feature vectors.


Table <ref>	presents linear regression models, including linear, interactions, robust, and stepwise linear models. We also evaluate regression Trees with three minimum leaf sizes, i.e., 4, 12, and 36 in this table.




Table <ref>	evaluates SVMs models through different Kernel functions and scales.  We consider kernel scales 0.5, 2, and 8 for fine, medium, and coarse Gaussian SVM, respectively.






Table <ref> shows GPR and Ensemble Trees models. The former is evaluated with squared exponential, Matern 5/2, exponential, and rational quadratic kernel functions. The boosted Trees and the Bagged Trees are examined for the latter.















Fig. <ref> displays predicted response versus subject numbers in BraTS'19. The predictions are accomplished with the stepwise linear regression model.



Fig. <ref> also illustrates predicted response based on three features. We removed age feature to evaluate the effect of this feature on OS task. Table <ref> compare RMSE with and without age feature for survival task. 


	

	
	
	
	
	

§ CONCLUSION

	
	
This paper provided a framework for the synthetic segmentation that translated FLAIR MR images into high-contrast synthetic MR ones for segmentation.  Synthesizing based on the GAN network empowers our model to decrease the number of real channels in multimodal brain tumor segmentation challenge 2019. We also implemented several regression models to predict the OS of each patient. We found that the stepwise linear model overwhelmed other traditional regression models in terms of RMSE. We also observed that patient age as a distinctive feature in the OS prediction tasks. 
	
	
	

§ ACKNOWLEDGMENT

	
	This work was supported partly by National Natural Science Foundation of China (Nos.61871274, 61801305, and 81571758), National Natural Science Foundation of Guangdong Province (No. 2017A030313377), Guangdong Pearl River Talents Plan (2016ZT06S220), Shenzhen Peacock Plan (Nos. KQTD2016053112 051497 and KQTD2015033016 104926), and Shenzhen Key Basic Research Project (Nos. JCYJ20170413152804728, JCYJ20180507184647636, JCYJ20170818142347 251, and JCYJ20170818094109846).
	
	
	
	
	
	
	
	
	
	
	
	splncs04
	
	
	
	
	
