Multi-view Cardiac Image Segmentation via Trans-Dimensional Priors
Abbas Khan Muhammad Asad Martin Benning Caroline Roney Gregory Slabaugh

















































[inst1]School of Electronic Engineering and Computer Science, Queen Mary University of London, UK
[inst2]Queen Mary's Digital Environment Research Institute (DERI), London, UK
[inst3]School of Biomedical Engineering and Imaging Sciences, King’s College London, UK
[inst4]Department of Computer Science, University College London, UK
[inst5]School of Engineering and Materials Science, Queen Mary University of London, UK

    
inst1,inst2]Abbas Khan
inst2,inst3]Muhammad Asad
inst4]Martin Benning
inst2,inst5]Caroline Roney
inst1,inst2]Gregory Slabaugh




We propose a novel multi-stage trans-dimensional architecture for multi-view cardiac image segmentation. Our method exploits the relationship between long-axis (2D) and short-axis (3D) magnetic resonance (MR) images to perform a sequential 3D-to-2D-to-3D segmentation, segmenting the long-axis and short-axis images. In the first stage, 3D segmentation is performed using the short-axis image, and the prediction is transformed to the long-axis view and used as a segmentation prior in the next stage. In the second step, the heart region is localized and cropped around the segmentation prior using a Heart Localization and Cropping (HLC) module, focusing the subsequent model on the heart region of the image, where a 2D segmentation is performed. Similarly, we transform the long-axis prediction to the short-axis view, localize and crop the heart region and again perform a 3D segmentation to refine the initial short-axis segmentation. We evaluate our proposed method on the Multi-Disease, Multi-View & Multi-Center Right Ventricular Segmentation in Cardiac MRI (M&Ms-2) dataset, where our method outperforms state-of-the-art methods in segmenting cardiac regions of interest in both short-axis and long-axis images. The pre-trained models, source code, and implementation details will be publicly available.






    < g r a p h i c s >





    
  * We propose a sequential 3D-to-2D-to-3D approach for multi-view cardiac image segmentation by effectively utilizing the trans-dimensional segmentation priors (TDSP), which transform a segmentation from one view into another and serve as guidance. 
    
  * The TDSP provides a robust anatomical reference at the network's input and encourages the network to produce anatomically plausible segmentation maps.
    
  * We also introduce a Heart Localization and Cropping (HLC) module to focus the segmentation on the heart region only. This strategy reduces the computation for the second and third-stage segmentation network and eliminates false positive predictions.
    
  * Extensive experiments are conducted to showcase the efficacy of the proposed pipeline utilizing the HLC module and TDSP, where our proposed method outperforms the state-of-the-art as well as methods on the M&Ms-2 challenge leaderboard. 

    

Cardiac MRI Image Segmentation Short-Axis Long-Axis Transformation Priors Sequential Segmentation









§ INTRODUCTION


Cardiovascular disease is the leading cause of death, with a yearly toll of 23.6 million lives due to heart disease and stroke globally <cit.>. This underscores the need to identify and treat cardiac disorders. Cardiologists have focused on early diagnosis as part of the clinical workflow <cit.>. Deep learning architectures have achieved a wide range of competencies for computational cardiac imaging <cit.>,<cit.>,<cit.> including segmentation <cit.>,<cit.>,<cit.>. 

Modern non-invasive medical imaging techniques, such as ultrasound, magnetic resonance imaging (MRI), and computed tomography, are widely used to capture detailed images of the structure and function of the heart and its associated vessels <cit.>. However, detection of disease and quantification often requires a laborious process of manual segmentation to identify the areas of the anatomy in these scans. Recent advances in artificial intelligence <cit.> are improving automation to segment a medical image into meaningful areas of interest. In the context of cardiac imaging, areas of interest include the left atrium, right atrium, left ventricle, right ventricle, and myocardium to diagnose different cardiac pathologies. Many image segmentation methods have been proposed, including active shape models <cit.>, active appearance models <cit.>, atlas-based methods <cit.>, convolutional neural network (CNN)-based approaches <cit.>,<cit.> including those with self-attention-based architectures <cit.>,<cit.>.  

Among the successful cardiac image segmentation methods, most rely on a single view, i.e., short-axis (SA) or long-axis (LA), where the segmentation is performed. However, capturing both SA and LA MR images is considered standard practice <cit.>, <cit.>, and segmentation of one view can be utilized to improve the segmentation of the other. Here, we propose a novel framework that performs accurate cardiac image segmentation by transferring the segmentation of one view to guide the segmentation of the other. Our proposed method sequentially utilizes the multi-view images. Despite being based on single encoder-decoder segmentation networks, the proposed pipeline still benefits from multi-view data.

fig1(d) depicts the overall architecture of the proposed pipeline. TriggerNet, which functions as a 3D segmentation model, generates the initial segmentation for the short-axis denoted as S_SA1.
Subsequently, utilizing the transformation parameters for the given volume, S_SA1 undergoes transformation to the LA view to produce a SA-to-LA map (SA2LAmap), a trans-dimensional segmentation prior. The SA2LAmap and LA image (I_LA) are fed as input to the Heart Localization and Cropping (HLC) module, resulting in a cropped I_LA and SA2LAmap containing only the heart region.  The SA2LAmap is input along with the cropped I_LA to the LA-SegNet model that generates a segmentation for the long-axis named S_LA.

We next refine the short axis segmentation S_SA1. S_LA is transformed to the SA view, resulting in the LA-to-SA map (LA2SAmap), another trans-dimensional segmentation prior. Here, we again use the HLC module to obtain cropped LA2SAmap, SA image (I_SA), and the TriggerNet output (S_SA1). Finally, the SA-SegNet utilizes these cropped outputs of the HLC module and generates the final segmentation for the short-axis named S_SA2.


In our proposed framework, integrating the segmentation from alternate views (SA to LA and LA to SA) acts as a segmentation prior, and provides HLC module guidance to remove the surrounding background regions and improve overall segmentation accuracy for the respective views. This framework enables LA-SegNet and SA-SegNet segmentation to outperform the existing state-of-the-art methods on the Multi-Disease, Multi-View & Multi-Center Right Ventricular Segmentation in Cardiac MRI (M&Ms-2) dataset's challenge <cit.>. The proposed framework efficiently utilizes the multi-view aspect of the M&Ms-2 dataset, as the challenge provides the images and labels of both views (LA and SA) for each instance, compared to the previous datasets M&Ms <cit.> and ACDC  <cit.>.



§ RELATED WORK

This section lists some of the most well-known deep learning-based segmentation architectures, including those unifying the power of CNN and self-attention-based mechanisms. We also detail existing cardiac image segmentation approaches, specifically from the M&Ms-2 dataset's challenge <cit.> leaderboard and subsequent publications leveraging the dataset. 
We note that our proposed pipeline can be implemented with any segmentation backbone, provided that the architecture can segment both 2D and 3D views.

UNet <cit.> revolutionized deep learning-based medical image segmentation by proposing a symmetric encoder-decoder architecture. The encoder part extracts the features from the image, and the decoder reconstructs the segmentation map, while the skip-connections help to propagate information across different stages.  The no-new-Net (nnUNet) <cit.> is built on UNet and proposed an automatically configurable segmentation architecture. It can configure data pre-processing, network design, and post-processing for many medical image segmentation datasets. An overview is provided in sec:nnunet. ResUNet <cit.> is an encoder-decoder architecture based on the UNet model and also incorporates knowledge of residual connections <cit.>, atrous convolutions <cit.>, and pyramid scene parsing (PSP) pooling <cit.>. Each convolution block is replaced with a residual block to achieve consistent training with the increased network depth, atrous convolutions help increase the receptive field, and PSP pooling enhances the network's performance by including background context information.

Inspired by the emergence of vision transformers <cit.> in computer vision regimes <cit.>, many hybrid architectures that utilize multi-head self-attention (MHSA) <cit.> have been proposed for medical image segmentation. TransUNet <cit.> is a UNet architecture that utilizes both CNN and self-attention. This includes a transformer-based encoder that extracts features from images and a CNN-based decoder that upsamples the encoded features. UTNet <cit.> is also a hybrid architecture integrating transformer and CNN for medical image segmentation. It proposes a revised MHSA mechanism to reduce the complexity of the model.  In addition, a hybrid layer utilizing CNN and revised MHSA is incorporated into the encoder and decoder stages. 
The Multi-Compound Transformer (MCTrans) <cit.> aims to combine rich features and semantic structures into multi-scale convolutional features using self-attention. The MCTrans transforms convolutional features as a sequence of tokens to perform intra- and inter-scale self-attention across multiple scales.
A multi-view and transformer-based architecture named Transfusion was proposed by <cit.> to correlate and fuse data coming from SA/LA views. It proposed Divergent Fusion Attention (DiFA), which combines features from different views using multi-scale self-attention.
Al Khalil et al. <cit.> proposed a three-stage approach: firstly, the region of the heart is detected using a regression model; secondly, a GAN-based augmentation technique is used for image synthesis to increase the diversity of the training data for segmentation tasks. More specifically, their approach generates more examples of pathologies to balance instances of pathological and normal cases. Lastly, the late-fusion segmentation approach combined with intensity transformations is utilized to generate the final segmentation map.


Sun et al. <cit.> utilized labels from the end-diastolic and end-systolic phases through an intensity-based image registration approach. These registered labels increase the size of the training set. Arega et al. <cit.> relied on the MRI-specific based, intensity, and spatial data augmentation techniques to improve the generalization and robustness of their segmentation models.  
In <cit.>, a multi-view SA-LA Network was proposed to simultaneously segment the RV blood pools in both the SA and LA views. It merged the bottleneck features from both the SA and LA and combined the labels of the left ventricle (LV) and myocardium (MYO) to generate a label that aids with contextual information to better segment the right ventricle (RV). Another multi-encoder-decoder network (xUnet) is proposed by <cit.> to simultaneously process the SA and LA views. It utilizes a pre-processing step where both views are centered and rotated to match their axes. A spatial transformer multi-pass feature pyramid (Tempera) <cit.> segments the RV in both SA and LA cardiac MR images. Tempera is based on the multi-scale feature pyramid network from <cit.> and transforms the SA features to LA via a geometric target spatial transformer. InfoTrans <cit.> proposed a nnUNet-based architecture, where the first 2D-nnUNet segments the LA views and then utilizes the LA prediction to crop the region of interest (ROI) from SA views.
The Refined Deep Layer Aggregation (RDLA) <cit.> proposed a two-stage 2D architecture, using DLA-34 stride-2 network <cit.> as the backbone. The LA and SA images are segmented independently, followed by a refinement step by utilizing the complementary information of another view along with the images.

Our proposed approach unites the strengths of <cit.>, <cit.>, and  <cit.>, as shown fig1  and introduces a sequential approach in which each network benefits from the previous one, without introducing additional parameters compared to other multi-stage approaches such as <cit.>. 
InfoTrans <cit.> performs information transition only from an LA to an SA network and utilizes the transformed SA map (LA2SAmap) only to extract the region of interest (ROI) from original SA images. Compared to this method, we introduce the transformation from LA to SA to obtain the LA2SAmap and from SA to LA to obtain the SA2LAmap. Additionally, we also take advantage of these transformation maps and utilize them as segmentation priors for anatomically plausible predictions.
The Tempera <cit.> architecture only transforms SA to LA and segments each view twice using a two-stage methodology. Compared to Tempera, the proposed pipeline performs a complete two-way transformation, using the SA2LAmap and LA2SAmap to achieve 3D-to-2D-to-3D segmentation. Furthermore, Tempera does not utilize the transformed map to localize and crop the heart region in intensity images, as proposed by the HLC module. Hence, both stages in Tempera utilize full-scale images. 
Compared to <cit.>, which is implemented using four 2D networks, the proposed refinement strategy uses three networks, where we use a 3D network for SA images to effectively exploit the 3D spatial context and a single 2D network can achieve state-of-the-art performance using the transformation from TriggerNet as a segmentation prior. In addition, in RDLA architectures, the second-stage networks have the same complexity as the first-stage networks without taking advantage of alignment and first-stage predictions. However, in the proposed setting, the second-stage networks, i.e., LA-SegNet and SA-SegNet, receive images with a lower in-plane spatial resolution (images containing only heart regions) and have fewer encoder-decoder stages, which results in reduced computational complexity, as shown in param (further details in ablation). Al Khalil et al. <cit.>  employed a regression-based neural network as a crucial component for heart region detection within SA and LA images. While practical, this regression-based neural network brings additional trainable parameters, thereby increasing the complexity of the overall pipeline. In contrast, our proposed HLC module achieves the exact heart region localization and cropping task as their regression-based model without introducing additional trainable parameters. Instead, the HLC module utilizes segmentation maps from the previous network and crops the heart regions, thus offering a streamlined alternative to the regression-based approach. Our contributions can be summarized as follows:

    
  * We propose a sequential 3D-to-2D-to-3D approach for multi-view cardiac image segmentation by effectively utilizing the trans-dimensional segmentation priors (TDSP), which transform a segmentation from one view into another and serve as guidance. 
    
  * The TDSP provides a robust anatomical reference at the network's input and encourages the network to produce anatomically plausible segmentation maps.
    
  * We leverage the TDSP and introduce a Heart Localization and Cropping (HLC) module to focus the segmentation on the heart region only. This strategy reduces the computation for the second and third-stage segmentation network and eliminates false positive predictions.
    
  * Extensive experiments are conducted to showcase the efficacy of the proposed pipeline utilizing the HLC module and TDSP, where our proposed method outperforms the state-of-the-art as well as methods on the M&Ms-2 challenge leaderboard. 




§ PROPOSED FRAMEWORK

fig1(d) depicts our proposed framework, where the pipeline starts with trigger network (TriggerNet), followed by the transformation of its predictions S_SA1 to the LA view, resulting in SA2LAMap. As a pre-processing step for SA to LA transformation, the header information of the original SA image (I_SA) is applied to S_SA1 to ensure the matching of all properties of the I_SA and S_SA1.
The SA2LAMap is used to remove the unrelated non-cardiac areas of the original LA image (I_LA) utilizing the HLC module and as input to the LA-SegNet through concatenation with the cropped I_LA as a segmentation prior. The output from LA-SegNet is restored to its original size, followed by copying all the metadata information from the I_LA to preserve the header information of the I_LA in S_LA.
The final segmentation for LA (S_LA) is transformed to the SA view using the I_SA to obtain LA2SAMap.

Following the same process, the HLC module utilizes the LA2SAMap and I_SA to localize and crop the heart in a full-scale image.
Here, we also cropped and concatenated the S_SA1 from the TriggerNet (further details are provided in the ablation in ablation). Finally, the S_SA2 is restored to its original size.

All three networks (TriggerNet, LA-SegNet, and SA-SegNet) are trained independently. The downstream tasks, such as LA-2-SA transformations and vice versa, and the HLC module are applied when the previous network outputs are available. However, at inference, the full framework is used sequentially to perform 3D-to-2D-to-3D segmentation output results for both the LA and SA images.

The following subsections will list the details of each step, segmentation networks, HLC module, and transformation process in the proposed pipeline shown in fig1.


 §.§ Segmentation Networks


Segmentation networks used within the proposed framework, i.e., TriggerNet, LA-SegNet, and SA-SegNet, are implemented using nnUNet <cit.>. The nnUNet is built upon the original U-Net architecture with modifications and improvements adopted for medical image segmentation tasks. It stands out from UNet due to its ability to configure the architecture and hyperparameters automatically during training. Moreover, nnUNet achieved the best results in the challenge cohort of  M&Ms-2 <cit.>, and the proposed pipeline is built using its architecture as a baseline. The nnUNet architecture has three major components: (i) Encoder, (ii)  Decoder, and (iii) Skip-Connections.




The encoder extracts features from the input data by gradually increasing the number of features while reducing the spatial dimensions as it goes deeper. Each encoder block consists of two consecutive convolutions with a kernel size 3 (3×3 for 2D/LA, and 3×3×3 for 3D/SA network). To reduce the spatial resolution, the features are again convolved with a kernel size of 3 and stride of 2. Each convolutional layer is followed by LeakyReLU activation and instance normalization.










The decoder reconstructs the segmentation map by progressively increasing the spatial dimension and reducing the number of features from the bottleneck layer. Each decoder block has two consecutive convolutions with a kernel size of 3, followed by a transpose convolution layer with a kernel size of 2 and stride 2. Similar to the encoder, each convolution layer is followed by LeakyReLU activation and instance normalization. The final convolution layer utilizes a sigmoid activation function with four kernels of size 1, where each kernel generates segmentation output for four classes, i.e., MYO, LV, RV, and background.

The skip-connections have been shown to improve segmentation methods <cit.> for medical image segmentation tasks, and hence, we utilize skip connections in our proposed architecture. These skip-connections copy and concatenate the features from the contracting path from the encoder to the expanding path in the decoder for a better gradient flow during backpropagation and to recover the lost spatial information.

The number of encoder-decoder blocks can be different for each segmentation network shown in fig1(d), and the corresponding computational complexity in param, depending upon the spatial dimension of the input data. The TriggerNet gets spatial dimension images of 64 × 192 × 192, and it has five downsampling (encoder's block) and upsampling blocks (decoder's blocks). The LA-SegNet network gets the cropped input of spatial size 128 × 128 and has the four encoder and corresponding decoder blocks. The spatial dimension of input data for SA-SegNet is 112 × 128 × 112, and it also has four blocks for downsampling and upsampling. We also trained the LA-SegNet network on LA images without utilizing the HLC module for the ablation studies mentioned in ablation. For this network, the nnUNet configures six encoder-decoder stages due to its large spatial dimension of 384 × 384.



 §.§ Heart Localization and Cropping (HLC) Module

The foreground-background imbalance of pixels has been a fundamental issue for accurately segmenting medical images <cit.>. The foreground pixels occupy a smaller proportion of the image than the background objects.
It can significantly degrade the segmentation performance by forcing the model to focus more on the background pixels due to their majority compared to the foreground pixels <cit.>. A common way of solving this issue is by designing the loss functions for segmentation, which can weigh the foreground pixels more than the background pixels <cit.>, <cit.>. However, in the proposed framework, this problem is solved intrinsically to some extent, as we cropped the original full-scale image using the segmentation prior, resulting in a reduction of background search space.

The segmentation networks utilizing full-scale images frequently confuse the background and cardiac tissue, resulting in considerable false positives. Using the proposed HLC module, the network only focuses on a smaller area containing heart tissues. Moreover, as the HLC module reduces the spatial resolution, the next stage network will run on a low-resolution image, leading to reduced computational complexity.

The HLC module is implemented by extracting the heart region containing the LV blood pool (LV), RV blood pool (RV), and left ventricular MYO from the original full-scale images. This can be achieved using the SA2LAmap for I_LA and either LA2SAmap or S_SA1 for I_SA.



To find the bounding box across three labeled regions (LV-blood pool, RV-blood pool, and left ventricular MYO) within a segmentation prior, we'll use transformation maps (SA2LAmap, LA2SAmap, or S_SA1) as binary masks, where nonzero values represent the regions of interest. To ensure that the cropped region occupies the entire region and does not miss any pixels of the foreground regions, we defined a parameter named margin, which can help to safeguard the edges and provide a margin that assists in preserving the entire heart in the cropped region, as shown in fig2.

We further confirmed that the obtained cropped region from the HLC module safely encloses the entire heart by cropping the ground truth, restoring the original size, and finding the Dice score between the original size and the restored ground truth. We ensured a Dice score of 1 and Hausdorff Distance (HD) of 0 for the resized ground truth segmentation against the original size ground truth. Empirically, we found that a margin of 15 pixels perfectly fits this purpose for all images in the training, validation, and test sets. We applied the same protocols to crop the intensity image and bring back the prediction from the cropped image to the original spatial dimension.



 §.§ The Transformation Process

The M&Ms-2 dataset is novel in terms of providing the images/labels of the LA view along with the SA view to give detailed information for the apical and basal slices of short-axis views <cit.>. We have utilized this information more efficiently in the proposed framework and generated the transformations for each axis. More specifically, we transformed one view's physical coordinates into the other view's image coordinates system and vice versa.

The trans-dimensional segmentaton prior SA2LAmap is obtained using the pseudo-code shown in Algorithm <ref>. The prediction from TriggerNet S_SA1 has different metadata information than the original I_SA. This metadata information is essential for the conversion between physical coordinates and image coordinate systems and includes additional information like image orientation, voxel size, and origin. We used the CopyInformation function from SimpleITK to inherit all the relevant metadata from the original SA image to the S_SA1. This ensured that S_SA1 was 


aligned correctly with the original I_SA and could be used to transform the coordinate systems with the original I_LA.

The physical coordinates of the I_LA are obtained from the image coordinate system, followed by finding the corresponding index in the S_SA1 and mapping these physical coordinates to the S_SA1 image coordinate system. Finally, it assigns the voxel value from the S_SA1 to the corresponding voxel in the SA2LAmap. In this way, the SA2LAmap is populated with all transformed values from S_SA1 to the SA2LAmap.

The trans-dimensional segmentation prior LA2SAmap is generated similarly, as shown in Algorithm <ref>. In this case, the metadata information from the original I_LA is copied to the S_LA from LA-SegNet.

The physical coordinates of the I_SA are extracted from its image coordinates system, followed by transforming this physical point to an index in the coordinate system of LA2SAmap.  Finally, each voxel value from the S_LA is copied to the corresponding location in the LA2SAmap.





 §.§ Implementation Details

The proposed architecture is implemented using a single NVidia A100 GPU with 40GB RAM. The SA and LA MRI scans are resampled to a voxel size of 1×1×1 mm^3. Dice loss and cross-entropy loss are used as loss functions to train the segmentation networks. Stochastic gradient descent is used as an optimizer with an initial learning rate of 0.01 and a Nesterov momentum of 0.99. We utilize a polynomial learning rate scheduler <cit.> with a weight decay of 0.0005 to decrease the learning rate after each training epoch.

All networks, i.e., TriggerNet, LA-SegNet, SA-SegNet, and the nnUNet baseline, are trained independently for 1000 epochs (nnUNet default), where each epoch has 250 training iterations. The pre and post-processing steps, such as LA-2-SA transformation and vice versa, and the heart localization and cropping are performed in succession after the previous network predictions are available. All three segmentation networks and their respective pre and post-processing steps are carried out sequentially in the inference phase.  

Different data augmentation techniques are applied during training to allow the networks to see a stream of distinct examples. Spatial transformations, including random rotation, scaling, and mirroring, provide distinct spatial perspectives from which the model can learn. Intensity adjustments, such as random brightness, contrast, and gamma variations, ensure the model's adaptability to varying acquisition settings. Additionally, additive zero-mean Gaussian noise is utilized to enhance stochasticity, and blurring techniques, such as Gaussian blur, contribute to the model's robustness against variations in image quality.


§ DATASET

The Multi-Disease, Multi-View, and Multi-Center Right Ventricular Segmentation challenge (M&Ms-2) was introduced in MICCAI 2021. The challenge focused on segmenting RV blood pools across cardiac imaging of multiple views and centers <cit.>,<cit.>. The data includes diverse images from three clinical centers in Spain utilizing nine scanners from three vendors, including Siemens, General Electric, and Philips. It includes instances having various LV and RV pathologies as well as healthy subjects. The labels are provided for three regions of interest, including (i) LV blood pools, (ii) RV blood pools, and (iii) left ventricular MYO. It contains 360 instances from two cardiac cycles, specifically the end-diastolic and end-systolic phases. The subjects are divided sequentially into 160 for training, 40 for validation, and 160 for testing, such that different patients are in each split. The validation and test set also includes patients with pathologies not included in the training set. For each individual, both SA and LA MR images are provided, having SA and LA 4-chamber views.


§ ABLATION STUDIES
 
We study the effectiveness of our algorithmic design via different ablation studies. In particular, we evaluate the effect of utilizing the SA2LAmap and LA2SAmap/S_SA1 as a segmentation prior and the HLC module under different settings.

The SA2LAmap can be used in two ways to boost the network's performance: (i) Localization and Cropping Guide for HLC module: To localize and crop the heart in original full-scale I_LA, and (ii) Segmentation Prior: As a Segmentation prior concatenated to the I_LA. tab1 lists the results of these experiments. 



Row 1 represents the baseline where we do not use the SA2LAmap. Rows 2 and 3 show the results where we are using the SA2LAmap for the HLC module or as a segmentation prior, and row 4 depicts the performance where we first localized and cropped the heart in the I_LA using the SA2LAmap and then concatenated the cropped image with the SA2LAmap. It can be seen that using either of the techniques (HLC module or segmentation prior) improves the results, which can be further validated with fig3.
In fig3, rows (a) and (b) depict that using the SA2LAmap either for the HLC module or as a segmentation prior can help remove the outliers (false positive predictions outside the area of interest). Rows (c) and (d) further confirm the usage of segmentation prior. The segmentation prior aids the network in generating anatomically plausible results. Thus, the SA2LAmap obtained from the S_SA1 helps to convert the invalid LA segmentation into close but correct shapes.



tab2 showcases the ablation results for SA segmentation under various settings. Row 1 uses neither the HLC module nor the segmentation priors and lists the results of the TriggerNet. Row 2 lists the results of SA-SegNet if we only utilize the predictions from the TriggerNet to localize and crop the heart in the original full-scale image and use it as a concatenated form along with the image. For row 3, we also utilized the LA2SAmap. Using the LA2SAmap as segmentation prior improves all metrics except the MYO Dice score; here, we argue that the collective structure segmentation of MYO has already been improved in terms of structure variance by S_SA1 as segmentation prior, and LA2SAmap helps further eliminate errors in the boundary regions of MYO as indicated by the improved HD score.



fig4 depicts the visual results of this ablation study. Row (a) and (b) validate that each segmentation prior (S_SA1 or LA2SAmap) helps to remove the outliers. However, the last two rows, (c) and (d), further confirm that using the LA-to-SA transformation as a segmentation prior can further improve the segmentation. Here, we argue that in the cases where the TriggerNet cannot produce the accurate segmentation map for any of the regions of interest, then LA-SegNet might have already segmented that instance accurately with valid anatomical shape prior, and we can transfer that knowledge to LA2SAmap and is used it as segmentation prior for the SA-SegNet.

param further advocates that the HLC module not only improves the segmentation performance but also decreases the overall computational requirements of the proposed pipeline. In param, all the networks using the HLC module to localize the heart region have fewer encoder-decoder stages than those utilizing full-scale images, resulting in fewer parameters and Giga Multiply-Accumulate Operations (GMac). This is because if the in-plane spatial resolution size for input images is 384 × 384, to get bottleneck features of in-plane spatial dimension = 6 × 6, the encoder needs to compress the features six times; however, if the heart is cropped and localized using HLC module, it will produce the images of in-plane spatial dimension = 192 × 192, and the encoder will perform the feature compression in five instances to obtain bottleneck features with in-plane spatial dimension of 6  × 6.



§ RESULTS AND DISCUSSION

We compared the results of the proposed approach with several state-of-the-art architectures using a five-fold cross-validation method as well as comparing against the challenge leaderboard using the provided validation and test set.



 


 

 

tab3 lists the comparison of the results with other segmentation architectures in terms of the Dice score and HD on a five-fold cross-validation split. The proposed method is able to produce robust segmentation across both SA and LA views. A substantial improvement can be seen in terms of reducing the HD score. This gain in performance comes from the fact that the proposed HLC module can remove the outliers efficiently, resulting in an accurate segmentation map compared to the full-scale image methods. fig5 further illustrates this effect.

Compared to full-scale prediction, the segmentation map generated by utilizing the proposed HLC module eliminates all the erroneous predictions outside the region of interest. We can further observe the variation in HD score for the predictions with and without the outliers. It can be seen that the difference in HD score for MYO and LV is very large compared to the RV because of the false positives.

tab4 provides results compared to the challenge leaderboard on the validation set and the test set subjects for RV segmentation. This further proves the generalization ability of the proposed approach to the unseen pathologies, as the validation and test set has two pathologies (Tricuspidal Regurgitation and Congenital Arrhythmogenesis) not available in the training set.

We also compared the proposed approach with other multi-stage segmentation methods, such as those mentioned in <cit.>, and tab5 lists the comparison results. Similar to <cit.>, the comparison is performed on the 200 subjects by combining the validation (40) and testing (160) examples, while the algorithm's development occurs solely based on the training data. The proposed method outperforms <cit.> for most of the evaluation metrics.



 §.§ Limitations and future work

The proposed pipeline is useful in the settings where we have cardiac MRI available for both LA and SA views. Clinically, both views are captured according to cardiac MRI acquisition protocol <cit.>, <cit.>; however, most of the publicly available datasets only provide short-axis cine MR images and their labels <cit.>. This unavailability of long-axis MR images can be considered a limitation of the proposed approach for datasets with single-view images only. However, we expect future cardiac MRI datasets to release more complementary information, such as both views, to take advantage of their relationship. We also encourage the research community to provide a 2-chamber and 3-chamber LA view, further exploiting the multi-view aspect of cardiac MR images.

Finally, we analyze that a medical image is influenced by the anatomical features of the image and the characteristics of the imaging equipment, such as vendor information, scanner type, etc. This inspires our future work, where we will aim to design more robust pipelines to incorporate metadata along with intensity images for segmentation tasks <cit.>. This will enable the segmentation networks to learn not only the appearance of images but also the specific interdependence of an image structure and image-capturing device.  




§ CONCLUSION

This paper proposes a cardiac image segmentation approach relying on the trans-dimensional segmentation priors between short-axis and long-axis views. We show that the method provides a substantial improvement in the accuracy of segmentation for cardiac images in LV, RV blood pools, and left ventricular MYO. The proposed approach effectively utilizes the relationship between the SA and LA views so that a segmentation in one view informs the segmentation in the other view. The transformed maps are used to localize and crop the heart region in the original full-scale image using the HLC module of the same axis and act as a segmentation prior to the other axis. The HLC module helps to remove the outliers and improves erroneous predictions. The segmentation prior encourages anatomically plausible segmentation maps. Extensive ablation studies are conducted to show the efficacy of proposed techniques, and the results are compared with the existing state-of-the-art methods utilizing the M&Ms-2 dataset.



§ ACKNOWLEDGMENTS

This work was funded through the mini-Centre for Doctoral Training in AI-based Cardiac Image Computing provided through the Faculty of Science and Engineering, Queen Mary University of London.  This paper utilised Queen Mary's Andrena HPC facility. Caroline Roney acknowledges funding from a UKRI Future Leaders Fellowship (MR/W004720/1). 
This work also acknowledges the support of the National Institute for Health and Care Research Barts Biomedical Research Centre (NIHR203330), a delivery partnership of Barts Health NHS Trust, Queen Mary University of London, St George’s University Hospitals NHS Foundation Trust and St George’s University of London.

elsarticle-num 













