title: Domain-Adversarial Learning for Multi-Centre, Multi-Vendor, and Multi-Disease Cardiac MR Image Segmentation
authors: Cian M. Scannell Amedeo Chiribiri Mitko Veta
text: 
  	
  	
  	
  	Domain-Adversarial Cardiac MR Segmentation
  	
  	
  	
  	
  	
  	C.M Scannell et al.
  	
  	
  	
  	School of Biomedical Engineering and Imaging Sciences, King's College London, London, UK
   Department of of Biomedical Engineering, Eindhoven University of Technology, Eindhoven,
  		the Netherlands
    Joint last authors 
  
  		cian.scannell@kcl.ac.uk
  	
  	Domain-Adversarial Learning for Multi-Centre, Multi-Vendor, and Multi-Disease Cardiac MR Image Segmentation
      Cian M. Scannell10000-0001-9240-793X Amedeo Chiribiri1,3 Mitko Veta2,3
      December 30, 2023
  ===========================================================================================================
  
  
  	
  	
  		Cine cardiac magnetic resonance (CMR) has become the gold standard for the non-invasive evaluation of cardiac function. In particular, it allows the accurate quantification of functional parameters including the chamber volumes and ejection fraction. Deep learning has shown the potential to automate the requisite cardiac structure segmentation. However, the lack of robustness of deep learning models has hindered their widespread clinical adoption. Due to differences in the data characteristics, neural networks trained on data from a specific scanner are not guaranteed to generalise well to data acquired at a different centre or with a different scanner. In this work, we propose a principled solution to the problem of this domain shift. Domain-adversarial learning is used to train a domain-invariant 2D U-Net using labelled and unlabelled data. This approach is evaluated on both seen and unseen domains from the M&Ms challenge dataset and the domain-adversarial approach shows improved performance as compared to standard training. Additionally, we show that the domain information cannot be recovered from the learned features.
  		
  	
  	
  	
  	
  	
  
  § INTRODUCTION
  
  	
  	The characterisation of cardiac structure and function is an important step in the diagnosis and management of patients with suspected cardiovascular disease. Cardiac magnetic resonance (CMR) is the method of choice for the non-invasive assessment of cardiac function and allows the accurate quantification of structural and functional parameters such as the chamber volumes and ejection fraction. A limiting factor for the routine analysis of these parameters in the clinic is that it requires the tedious manual delineation of the anatomical structures.
  	
  	Deep learning has become the state-of-the-art approach for image segmentation and convolutional neural network (CNN)-based approaches have shown huge potential for the automated analysis of cardiac magnetic resonance (CMR) images <cit.>. Ruijsink et al. <cit.> recently showed that it is further possible to automatically compute a wide range of advanced indices of cardiac function, including shape, motion, and strain, from short-axis cine CMR images. This is an important step as it makes it feasible to analyse large numbers of patients and it reduces the dependency of the analysis on the operator <cit.>.
  	
  	However, the aforementioned studies are all limited by the homogeneity of the data that was used. They were either single/few centre studies or they used publicly available databases such as the UK Biobank which have uniform imaging protocols and few pathological cases. As such, the datasets do not match the variability seen in clinical practice and although the studies report good performance on their data, it is not guaranteed that they will generalise well to real-world settings. Images acquired using different scanners can have widely-varying levels of signal, noise, and contrast. Images acquired at different centres may be planned differently resulting in differing locations of the heart in the images, and distinct cardiovascular diseases can alter the shape of the heart. Some of these variations are shown in Fig. <ref>. These variations introduce a so-called domain shift, a change between the distribution of training data and the distribution of the data that the model is being applied to <cit.>.
  	
  	
  	
  	Since deep learning models are known to not generalise well in the presence of such a domain shift <cit.>, this leads to the need for techniques for domain adaption. The most simple technique for domain adaption is use of data augmentation to try to account for the variability between domains during training <cit.>. It is, however, difficult to account for all possible variations in this manner. Other approaches include fine-tuning previously learned weights on a new domain and, more recently, conditional generative models, such as cycleGAN <cit.>, have been used to translate images from a new domain to the domain of the model. However, training data would be required from the new domain for these approaches and thus they cannot be applied to completely unseen domains.
  	
  	Ganin et al. <cit.> proposed an approach to the domain-adversarial training of neural networks based on the assumption that for a model to generalise well to new domains the features learned by the model must be unable to discriminate between domains. This approach has been recently applied to domain adaption in histological images and shows promising results <cit.>. In this work, domain-adversarial training is used for cine CMR segmentation and is shown to learn features that are domain invariant and generalise well to new unseen domains.
  	
  	
  	
  
  § MATERIALS AND METHODS
  
  	
  
   §.§ Dataset
  
  	The dataset for this study was provided by the Multi-Centre, Multi-Vendor & Multi-Disease (M&Ms) Cardiac Image Segmentation Challenge[<https://www.ub.edu/mnms/>] <cit.>. The datasets consisted of 350 patients with a mix of healthy controls and patients with hypertrophic and dilated cardiomyopathies. It was acquired at six different clinical centres using MRI scanners from four different vendors. From this dataset 150 annotated patients from two different MRI vendors were made available for training as well as 25 patients from a third MRI vendor without annotations. There were 40 patients, 10 from each vendor, for validation and 160 patients, 40 from each vendor, for testing. These images were not available but results could be obtained from the challenge organisers. Since the test set for the challenge could only be used once, the comparison of our approach to the baseline is performed on this challenge validation set and 32 patients from the training set were used for internal validation. Expert annotations are provided for the left (LV) and right ventricle (RV) blood pools, as well as for the left ventricular myocardium (MYO).
  	
  	
  
   §.§ Domain-Adversarial Learning
  
  	Domain-adversarial learning attempts to improve the generalisation of neural networks by encouraging them to learn features that do not depend on the domain of the input. In this situation, as well as learning a network for the segmentation task, we concurrently learn a classifier which attempts to discriminate between the input domains using the activations of the segmentation network. An adversarial training step is then introduced which is a third training step after the optimisation of the segmentation network and the optimisation of the domain discriminator. This adversarial step uses a gradient reversal to update the weights of the segmentation network to maximise the loss of the domain discriminator and thus prevents the domain information being recovered from the learned representations of the segmentation network. The three optimisation steps, using a standard (stochastic) gradient descent algorithm would be: 
  	
      θ_S←θ_S - λ_S ∂ℒ_𝒮/∂θ_S
  
  	
      θ_D←θ_D - λ_D ∂ℒ_𝒟/∂θ_D
  
  	
      ϑ_S←ϑ_S + αλ_S ∂ℒ_𝒟/∂ϑ_S
  
  	where θ_S, λ_S, & ℒ_𝒮 and θ_D, λ_D, & ℒ_𝒟 are the parameters, learning rate, and loss function of the segmentation network and domain discriminator, respectively. ϑ_S⊂θ_S are the parameters of the convolutional layers of the segmentation network and α∈ [0,1] controls the strength of the adversarial update. The proposed training pipeline is illustrated in Fig. <ref>.
  	
  	
  	
  	
  	
  
   §.§ Implementation Details
  
  	Our TensorFlow implementation and trained model weights are provided on github[<https://github.com/cianmscannell/da_cmr>].
  	
  	The baseline model used in this work was a 2D U-Net model <cit.>. The model consists of four down-sampling blocks followed by max-pooling with each down-sampling block being made of up two convolution blocks containing a convolutional layer, batch normalisation, and ReLU activation. 
  	
  	The domain discriminator takes as input the activations from the penultimate and lowest resolution layers of the U-Net. The model consists of 4 convolutional layers with ReLU activations, batch normalisation, and max-pooling and three fully-connected layers also with ReLU activation. For the purpose of this work, different MRI scanner vendors were considered as distinct domains.
  	
  	Images are resampled to a uniform resolution of 1.25 × 1.25  and the network is trained on images cropped to 192 × 192 pixels. Pixel intensity values are normalised to the range [0,1] before data augmentation. In this work, we use extensive spatial augmentation consisting of stochastic scaling, translation, rotation, and B-spline deformations using gryds <cit.>. Intensity augmentation is performed by adding Gaussian noise and random intensity shifts. The intensities are further normalised using contrast limited adaptive histogram equalisation.
  	
  	The segmentation network is trained with batch size of 16 to optimise the sum of (1 - DSC) and the cross-entropy, where DSC is the Dice similarity coefficient. The domain discriminator is trained with a batch size of 20 and the cross-entropy loss. Optimisation is performed using the ADAM optimiser in both cases. Early-stopping is used on the domain classification accuracy to chose a model that has the least dependence on domain knowledge after the segmentation accuracy has plateaued.
  	
  	Images from the two annotated domains are used to train the segmentation networks while images from all three available domains are used to train the domain discriminator and compute the adversarial update, including the unlabelled images, as no ground-truth segmentations are required. Since the domain-adversarial training is unstable <cit.>, the models are trained in stages. For the first 150 epochs, the segmentation U-Net is trained only with a learning rate of 1e-4. For the next 150 epochs, the domain discriminator is trained only with a learning rate of 1e-3. This initial training without the segmentation network updating allows it to achieve a baseline level of performance without its inputs (the activations of the segmentation network) changing. After this, both networks are trained together, including the adversarial update. The weighting of the adversarial update increases linearly from 0 to 1 over the next 150 epochs. The learning rate for the segmentation network (1e-3) is higher than the learning rate for the domain discriminator (1e-4) to allow the segmentation network to learn more quickly than the domain discriminator. At test time, predictions are made in a sliding window fashion over a 3x3 grid of patches with input size (192x192).
  	
  	This is compared to a baseline model trained in the exact same way, using the same intensity and data augmentation, except for the domain-adversarial update. The comparison was made using the nonparametric Mann-Whitney U test in SciPy <cit.>.
  	
  	
  
  § RESULTS
  
  	The loss and accuracy curves are shown for the domain adversarial training in Fig. <ref>.
  	 
  	The mean (SD) Dice similarity coefficient for the baseline model is 0.8 (0.18), 0.76 (0.13) and 0.77 (0.2) for the LV, MYO, and RV, respectively. The equivalent scores for the domain-adversarial model are 0.9 (0.07), 0.83 (0.05), and 0.87 (0.07). The Hausdorff distances for the three classes, in , are 17.6 (12.8), 26.3 (27.7), and 19.0 (12.1) for the baseline model and 12 (15.2), 17.4 (19.8), and 18.4 (33.4) for the domain-adversarial model. Figure <ref> further compares the baseline and domain-adversarial models showing the Dice for all three classes and how it varies across domains. The final test set results for our approach give Dice scores of 0.88 (0.1), 0.8 (0.09), and 0.84 (0.14) and Hausdorff distances () of 14.54 (19.12), 17.36 (20.86), and 17.51 (19.01) for the LV, MYO, and RV, respectively. There is no performance drop-off across domains and a summary of all performance metrics is shown in the Table. <ref>
  	
  	
  	The t-SNE embeddings of the learned representations of the baseline model are compared to the domain-adversarial learning approach in Fig. <ref>. With the baseline training there are clear clusters in the learned features indicating that they are domain dependent. For the domain-adversarial approach the learned features are more widely distributed in the embedding space, including for the domain not used for the training of the segmentation network (pink squares). Figure <ref> shows the output segmentations for example images from one of the unseen domains, comparing the baseline model to the domain-adversarial approach.
  	
  	
  	
  
  
  
  
  	
  
  § DISCUSSION
  
  	This paper deals with the challenge of the generalisation of deep learning models across different data domains. In particular, it deals with the segmentation of short-axis cine CMR images which vary significantly across MRI scanner vendors. 
  	
  	We have demonstrated that domain-adversarial learning can improve the generalisation of a U-Net model trained to segment short-axis cine CMR images. We trained the model using annotated data from two vendors and incorporated the domain information using unannotated data from a third vendor. The model can be applied on data from unseen domains (i.e from different MRI vendors) with no significant performance decrease. Although baseline training includes intensity normalisation and contrast normalisation through the use of histogram equilisation, lower performance is found on data from unseen domains. The comparison over the whole validation set shows statistically significant improvements using domain-adversarial learning, for the segmentation of all structures. This indicates that there are further variations across vendors, in addition to intensity and contrast, that the network relies on for the segmentation. The improved performance of the domain-adversarial approach shows that it can avoid over-fitting to this domain information and that it can learn more general, domain-invariant features. The performance of the baseline model is also worse on the training domains. This is likely because even data from the same vendor can be from different clinical centres which in itself may constitute a domain shift and because the domain-adversarial update has a regularising effect on training.
  	
  	Domain-adversarial learning has the benefit, over alternative domain-adaption strategies, of not requiring any data from the domains that the model is being applied on and not requiring any re-training. It also allows the incorporation of data without annotations in the domain discriminator and adversarial update so that unlabelled data can still be used to encourage domain invariance.
  	
  	As a limitation, the domain-adversarial training of neural networks is an unstable process. It relies on the ad-hoc hyper-parameter tuning more than convention training and work to address this would be needed for the technique to be used more commonly. Though the dataset used does include some pathological cases, these are not with a wide range of conditions and it remains to be determined if the domain-adversarial approach can still adapt to patients with different conditions that alter the geometry of the heart, such as ischaemic heart disease.
  	
  	In conclusion, domain-adversarial learning improves the ability of deep learning approaches for CMR image segmentation to generalise to unseen data from new domains. It does not require data from the new domains or to fine-tune/adapt the network. 
  	
  	
  
  § ACKNOWLEDGEMENTS
  
  	The authors of this paper declare that the segmentation method they implemented for participation in the M&Ms challenge has not used any pre-trained models nor additional MRI datasets other than those provided by the organizers. 
  	
  	The authors acknowledge financial support from the King’s College London & Imperial College London EPSRC Centre for Doctoral Training in Medical Imaging (EP/L015226/1); Philips Healthcare; and The Centre of Excellence in Medical Engineering funded by the Wellcome Trust and EPSRC under grant number WT 203148/Z/16/Z.
  	
  	
  	
  	
  	
  	8
  		
  		Bernard2018
  		Bernard, O. et al.: Deep Learning Techniques for Automatic MRI Cardiac Multi-Structures Segmentation and Diagnosis: Is the Problem
  		Solved? IEEE Trans. Med. Imaging 37(11), 2514–2525 (2018).
  		
  		Bai2018
  		Bai, W. et al. Automated cardiovascular magnetic resonance image analysis with fully convolutional networks. J. Cardiovasc. Magn. Reson. 20(65), 1–12 (2018).
  		
  		Tao2019
  		Tao, Q. et al. Deep Learning–based Method for Fully Automatic Quantification of Left Ventricle Function from Cine MR Images: A Multivendor, Multicenter Study. Radiology 290(1), 81–88 (2019).
  		
  		Scannell2020
  		Scannell, C. M. et al. Deep‐Learning‐Based Preprocessing for Quantitative Myocardial Perfusion MRI. J. Magn. Reson. Imaging 51(6), 1689–1696 (2020).
  		
  		
  		Ruijsink2020
  		Ruijsink, B. et al. Fully Automated, Quality-Controlled Cardiac Analysis From CMR: Validation and Large-Scale Application to Characterize Cardiac Function. JACC Cardiovasc. Imaging 13(3), 684–695 (2020).
  		
  		Villa2018 
  		Villa, A. D. M. et al. Importance of operator training and rest perfusion on the diagnostic accuracy of stress perfusion cardiovascular magnetic resonance. J. Cardiovasc. Magn. Reson. 20(1), 74 (2018).
  		
  		Dou2018
  		Dou, Q. et al.: Unsupervised Cross-Modality Domain Adaptation of ConvNets for Biomedical Image Segmentations with Adversarial Loss. In: Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18 691–697 (2018). 
  		
  		Chen2020
  		Chen, C. et al. Improving the Generalizability of Convolutional Neural Network-Based Segmentation on CMR Images. Frontiers in Cardiovascular Medicine  7, 105 (2020).
  		
  		Yan2020
  		Yan, W. et al. MRI Manufacturer Shift and Adaptation: Increasing the Generalizability of Deep Learning Segmentation for MR Images Acquired with Different Scanners. Radiol. Artif. Intell. 2, e190195 (2020).
  		
  		Ganin2016
  		Ganin, Y. et al. Domain-Adversarial Training of Neural Networks. J. Mach. Learn. Res. 17(59), 1–35 (2016).
  		
  		Lafarge2019
  		Lafarge, M. W., Pluim, J. P. W., Eppenhof, K. A. J., Veta, M. Learning Domain-Invariant Representations of Histological Images. Front. Med. 6, 162 (2019).
  		
  		Campello
  		Campello, Víctor M. et al.: Multi-Centre, Multi-Vendor & Multi-Disease Cardiac Image Segmentation. (In preparation).
  		
  		Ronneberger2015
  		Ronneberger, O., Fischer, P., Brox, T. U-Net: Convolutional Networks for Biomedical Image Segmentation. In: Navab, N., Hornegger, J., Wells, W. M. ,Frangi, A. F. (eds.) MICCAI 2015 LNCS 9351 234–241 Springer (2015).
  		
  		Eppenhof2019
  		Eppenhof, K. A. J., Pluim, J. P. W. Pulmonary CT Registration through Supervised Learning with Convolutional Neural Networks. IEEE Trans. Med. Imaging 38(5), 1097–1105 (2019).
  		
  		Virtanen2020
  		 Virtanen, P. et al. SciPy 1.0: fundamental algorithms for scientific computing in Python. Nat. Methods 17, 261–272 (2020).
  		
  		
  	
  