REAL-TIME AUTOMATIC FETAL BRAIN EXTRACTION IN FETAL MRI BY DEEP
LEARNING
Seyed Sadegh Mohseni Salehi?† , Seyed Raein Hashemi?‡ , Clemente Velasco-Annis? ,∗
Abdelhakim Ouaalam? , Judy A. Estroff? , Deniz Erdogmus† , Simon K. Warfield? , Ali Gholipour?
?

arXiv:1710.09338v1 [cs.CV] 25 Oct 2017

†

Computational Radiology Laboratory, Boston Children’s Hospital, and Harvard Medical School
Electrical and Computer Engineering Department, Northeastern University, Boston, MA, USA
‡
Computer and Information Science Department, Northeastern University, Boston, MA, USA

ABSTRACT
Brain segmentation is a fundamental first step in neuroimage
analysis. In the case of fetal MRI, it is particularly challenging and important due to the arbitrary orientation of the fetus,
organs that surround the fetal head, and intermittent fetal motion. Several promising methods have been proposed but are
limited in their performance in challenging cases and in realtime segmentation. We aimed to develop a fully automatic
segmentation method that independently segments sections of
the fetal brain in 2D fetal MRI slices in real-time. To this end,
we developed and evaluated a deep fully convolutional neural
network based on 2D U-net and autocontext, and compared it
to two alternative fast methods based on 1) a voxelwise fully
convolutional network and 2) a method based on SIFT features, random forest and conditional random field. We trained
the networks with manual brain masks on 250 stacks of training images, and tested on 17 stacks of normal fetal brain images as well as 18 stacks of extremely challenging cases based
on extreme motion, noise, and severely abnormal brain shape.
Experimental results show that our U-net approach outperformed the other methods and achieved average Dice metrics
of 96.52% and 78.83% in the normal and challenging test
sets, respectively. With an unprecedented performance and
a test run time of about 1 second, our network can be used to
segment the fetal brain in real-time while fetal MRI slices are
being acquired. This can enable real-time motion tracking,
motion detection, and 3D reconstruction of fetal brain MRI.
Index Terms— U-net, fetal brain segmentation, fetal
MRI, brain extraction, convolutional neural network
1. INTRODUCTION
Fetal magnetic resonance imaging (MRI) is clinically performed to evaluate the fetus in cases with suspicious or detected abnormalities on prenatal ultrasound [1]. Fetal MRI
∗ This study was supported in part by NIH grants R01 EB018988, R01
EB013248, and R03 DE022109; the McKnight Foundation, and the Brain
and Behavior Research Foundation. The trained model associated with this
work is available at bitbucket.org/bchradiology/u-net.

has also been instrumental in monitoring and characterizing
fetal brain development. Nonetheless, MRI is very susceptible to motion and fetuses move significantly during MRI
scans. To minimize the effects of motion, fetal MRI is performed through fast 2D snapshot imaging; i.e. fast 2D slices
are acquired one after each other to form a stack of slices.
Each slice acquisition, which takes between 1-2 seconds provides a good-quality image of a section of the anatomy, but
the stack of slices does not represent coherent 3D anatomic
boundaries due to inter-slice motion artifacts [2].
Recent advances in fetal MRI processing have shown that
3D images of the fetal brain can be reconstructed through
inter-slice motion correction and super-resolution volume reconstruction [3, 4, 5, 6]. These techniques assume a rigid
model for fetal head motion thus require manually or automatically detected brain masks. Accurate brain extraction has
shown to improve inter-slice motion correction and volume
reconstruction [6, 7]. The need for brain masks has motivated
a series of studies into fetal brain extraction in fetal MRI.
Keraudren et al. used bundled SIFT (bSIFT) features to
detect the brain in fetal MRI [8] and segmented the brain using an approach based on random forests and conditional random field (RF-CRF) [7]. Taimouri et al. [9] proposed a block
matching approach to simultaneously detect a bounding box
around the brain and match the orientation of the brain to a
template. Due to a search in the space of possible orientations, this technique was computationally very expensive.
Tourbier et al. [10] proposed an atlas-based fetal brain segmentation approach that required a predefined bounding box
around the brain. This approach was also very time consuming as it relied on deformable registration to multiple atlases.
For fast fully automatic brain segmentation, Khalili et al. [11]
recently proposed a voxelwise convolutional neural network
adopted from [12]. They augmented their neural network output with post-processing based on the connected components
algorithm to generate brain masks. All of the above methods
reported average Dice metrics slightly above 90%.
To further improve segmentation accuracy, extend its
use, and achieve real-time performance by avoiding any

post-processing in this work, we have adopted a deep fully
convolutional neural network based on the U-net [13], which
has recently shown great performance in 3D brain image
segmentation [14]. While the technique in [14] focused on
segmenting the brain in 3D reconstructed images, our goal
here is to segment brain sections independently on original
2D fetal MRI slices, which is much more challenging due
to motion and 2D appearance of brain sections. To this end,
we trained our network using manually-drawn brain masks
on 250 fetal MRI stacks from 26 subjects, and compared its
performance on typical and extremely challenging test sets
against a 3-pathway voxelwise network plus post processing
as suggested in [11], as well as the bSIFT-RF-CRF method
based on [7]. Experimental results show that our proposed
method based on U-net outperforms the other methods and
achieves a test run time of less than 1 second, which makes it
particularly attractive and useful for real-time application.
2. METHODS
2.1. A 2D U-net architecture
We designed and evaluated our first fully convolutional network [15, 16] based on the U-net architecture [13] and AutoNet [14]. We describe the details of this network here.
The U-net style architecture consists of a contracting path
and an expanding path. The contracting path contains padded
3 × 3 convolutions followed by ReLU non-linear layers. A
2 × 2 max pooling operation with stride 2 is applied after
every two convolutional layers. After each downsampling by
the max pooling layers, the number of features is doubled. In
the expanding path, a 2 × 2 upsampling operation is applied
after every two convolutional layers, and the resulting feature
map is concatenated to the corresponding feature map from
the contracting path. At the final layer a 1×1 convolution with
linear output is used to reach the feature map with a depth
equal to the number of classes (brain or non-brain tissue).
The output layer in the network consists of 2 planes (one
per class). We applied softmax along each pixel to form the
loss. We did this by reshaping the output into a width ×
height × 2 matrix and then applying cross entropy. To balance the training samples between classes we calculated the
total cost by computing the weighted mean of each class. The
weights are inversely proportional to the probability of each
class appearance, i.e. higher appearance probabilities led to
lower weights. Cost minimization on 100 epochs was performed using ADAM optimizer [17] with an initial learning
rate of 0.0001 multiplied by 0.9 every 2000 steps. The training time for this network was approximately three hours on a
workstation with an Nvidia Geforce GTX1080 GPU.
We did not use 3D U-net [18, 19] here because 1) interslice motion and motion-induced artifacts lead to structural
boundaries that are incoherent in 3D, 2) slices are occasionally disrupted by fast motion and appear dark, and 3) slice

thickness is usually much larger than the in-plane resolution.
We aimed to segment brain sections on slices independently
to detect good-quality brain structures and motion-corrupted
slices, therefore we chose a 2D U-net architecture rather than
its 3D counterpart.
2.2. A voxelwise approach
We also designed and evaluated a voxelwise network architecture [14] adopting a 3-pathway method and post processing
based on the connected components algorithm both suggested
in [11], as well as a morphological closing operation as the final step. To accelerate testing, we used fully convolutional
layers instead of the fully connected layers used in [11]. We
describe the details of this network here.
The voxelwise architecture consists of three fully convolutional paths. Each of these pathways contains four convolutional layers followed by a ReLU nonlinear function and
batch normalization. In the first layer, 24 kernels of size 5 × 5
were used for the patch sizes of 15 × 15 and 25 × 25, and
7 × 7 kernels for the patches of size 51 × 51. The second
layer contained 32 kernels of sizes 3 × 3, 3 × 3 and 5 × 5 on
patches respectively. The third layer was designed to have 48
kernel sizes of 3 × 3 for all patches. In the proposed architecture, fully convolutional layers were used instead of fully
connected layers [20] to achieve much faster testing time,
as the whole image can be tested in a network with convolutional layers while voxels are tested in a network with
fully connected layers [14]. This fully convolutional method
does not use any pre-processing method like brain localization, and is much faster in training and testing compared to
fully connected methods such as [11]. We also argue that doing a post processing like the 3D connected components is
not necessarily a more efficient and accurate way than using
pre-processing procedures like localization.
We used 30,000 samples per image and for each sample
used three 2D patches with different sizes of 51 × 51, 25 × 25
and 15 × 15 to collect local and global context information.
These patch sizes were shown to perform best in a 2.5D network for 3D brain image segmentation [14]. The training
process took 2 hours on an Nvidia Geforce GTX 1080 GPU,
passing 3 epochs using ADAM optimizer [17] with an initial
learning rate of 0.00005. Early stopping was used to prevent
overfitting and decrease the time of unnecessary training.
3. RESULTS
Data for this study was obtained from fetal MRI scans performed on 3T Siemens Skyra scanners at Boston Children’s
Hospital, where the Institutional Review Board committee approved the imaging protocols. For each subject multiple single shot fast spin echo (SSFSE) images were acquired with
in-plane resolution of 1 to 1.25 mm, and slice thickness of
2 to 4 mm. The gestational age of the fetuses at the time of

Table 1. Comparing Dice, sensitivity and specificity for normal test set, showing 5% higher average Dice score for U-net
compared to the voxelwise method after post processing (PP).
Method
Our 2D U-net
Voxelwise-PP
Voxelwise
SIFT-RF-CRF

Dice
96.52%
91.66%
84.37%
83.94%

Sensitivity
94.60%
90.64%
76.30%
74.50%

Specificity
99.92%
99.62%
99.77%
99.80%

Time
0.9s
45s
30s
60s

dicted brain mask P with the ground truth mask R, as
2|P ∩R|
2T P
D = |P
|+|R| = 2T P +F P +F N , where T P , F P , and F N
are the true positive, false positive, and false negative rates,
N
respectively. We also evaluate specificity, T NT+F
P , and senTP
sitivity, T P +F N , of the network.
Table 1 shows the results of the U-net based method
to those of the voxelwise approach (before and after postprocessing) and the bSIFT-RF-CRF method proposed in [7].
Our U-net based method shows the highest Dice coefficient,
with an increase of about 5% compared to the post processed
voxelwise approach. The average testing time for U-net was
0.915 second per stack (less than 50 ms per slice). This is
considered real-time since each SSFSE slice takes about 1-2
seconds to acquire on the MRI scanner.

Fig. 1. Boxplots of evaluation scores (Dice, sensitivity, specificity) for 2 test sets (normal and challenging cases). The deep
learning methods worked for all stacks for all cases, but the
bSIFT-RF-CRF method failed in 11 out of 18 stacks (3 out of
5 challenging cases) so the challenging test results for bSIFTRF-CRF are based on 7 stacks from 2 cases only. The U-net
based method shows the best scores.
scans was between 22 to 38 weeks (mean=29, stdev=5). In total, 285 stacks were selected from 33 fetal MRI sessions. The
fidelity of data selection was reviewed by a radiologist with
specialty expertise in fetal MRI. A trained research assistant
precisely and manually segmented the fetal brain in each slice
of all stacks. These were considered ground truth segmentations for training and testing. A total of 250 stacks from 26
sessions were selected as training data, 17 stacks from 2 typical sessions were selected as normal test data, and 18 stacks
from 5 very challenging sessions were selected as challenging test data. The challenging sessions involved cases with
extreme motion, a case with very abnormal brain shape due
to dural sinus fistula, and a case with extremely noisy images.
To evaluate and compare performance of 3 fast methods, Dice overlap coefficient was used to compare the pre-

Figure 1 shows boxplots of Dice, sensitivity and specificity of all the analyzed methods for normal and challenging
test datasets. In Figure 2, we demonstrate slices of the results
of the U-net and voxelwise methods for three cases: a case
from the normal test set; a case with severe brain abnormality; and three adjacent slices of a case with extreme motion as
well as an out-of-plane view of these images. Although there
are lots of inter-slice motion, the U-net based method was
capable of segmenting the brain quite accurately. The upsampling layers and concatenations in U-net helped effectively
learning multi-scale information on 2D slices. Overall, both
quantitative and qualitative evaluation results indicate that Unet performed significantly better than the 3-pathway voxelwise method in brain segmentation. We note that the bSIFTRF-CRF method failed in 11 out of 18 challenging test stacks
in the brain detection step presumably due to the failure of
a RANSAC process to reject the influence of outliers. These
cases with severely abnormal brain shape and extreme motion
pose challenges for such statistically-driven methods.
The original voxelwise Dice score on the normal test set
was 84.37%±0.05. After applying the connected components
algorithm we achieved a Dice score of 85.29% ± 0.03; and
finally after a morphological closing operation with a radius
of 5 voxels, we achieved average Dice score of 91.66%±0.02,
sensitivity of 90.64%±0.03 and specificity of 99.62%±0.002
as shown in Table 1. The average testing time for each slice
was about 1 second, and the average time for post processing
was about 15 seconds per stack.

Fig. 2. Predicted brain masks overlaid on original fetal MRI for a normal case (top row), an abnormal case (second row), and a
case with extreme motion (last four rows showing 3 consecutive slices and an out-of-plane view of the same stack); using U-net
(first column), the voxelwise method (second column), voxelwise after post processing (third column), and the manually drawn
ground truth (last column). The U-net approach performed much better than the voxelwise method. Note that U-net detected
part of the brain in a motion-corrupted scan which was not segmented by the expert due to perceived motion corruption.
4. CONCLUSION
We developed and tested a 2D U-net and a voxelwise fully
convolutional network to automatically segment fetal brain in
fetal MRI. For the voxelwise approach, we used post processing methods that significantly boosted our prediction accuracy on normal cases; but the U-net approach achieved much
better results on both normal and challenging test sets with
a testing time of about 1 second per stack without any postprocessing. The accuracy of our fast methods in segmenting the intracranial region of fetal MRI can lead to improved
methods of motion detection and correction, brain segmentation, and reconstruction which are subjects of ongoing work.

Table 2. Comparing the evaluation scores for the challenging
test set, showing 12% higher average Dice score for U-net
compared to the voxelwise method with post-processing.
Method
U-net
Voxelwise-PP
Voxelwise
SIFT-RF-CRF*

Dice
78.83%
66.87%
66.32%
64.95%

Sensitivity
71.97%
58.66%
57.21%
84.19%

Specificity
99.82%
99.76%
99.68%
98.02%

5. REFERENCES
[1] Orit A Glenn, “MR imaging of the fetal brain,” Pediatric
radiology, vol. 40, no. 1, pp. 68, 2010.
[2] Ali Gholipour, Judith A Estroff, Carol E Barnewolt,
Richard L Robertson, P Ellen Grant, Borjan Gagoski,
Simon K Warfield, Onur Afacan, Susan A Connolly,
Jeffrey J Neil, Adam Wolfberg, and Robert V Mulkern,
“Fetal MRI: a technical update with educational aspirations,” Concepts in Magnetic Resonance Part A, vol. 43,
no. 6, pp. 237–266, 2014.
[3] Ali Gholipour, Judy A Estroff, and Simon K Warfield,
“Robust super-resolution volume reconstruction from
slice acquisitions: application to fetal brain mri,” IEEE
transactions on medical imaging, vol. 29, no. 10, pp.
1739–1758, 2010.
[4] Maria Kuklisova-Murgasova, Gerardine Quaghebeur,
Mary A Rutherford, Joseph V Hajnal, and Julia A Schnabel, “Reconstruction of fetal brain MRI with intensity
matching and complete outlier removal,” Medical image
analysis, vol. 16, no. 8, pp. 1550–1564, 2012.
[5] Bernhard Kainz, Markus Steinberger, Wolfgang Wein,
Maria Kuklisova-Murgasova, Christina Malamateniou,
Kevin Keraudren, Thomas Torsney-Weir, Mary Rutherford, Paul Aljabar, Joseph V Hajnal, , and Daniel Rueckert, “Fast volume reconstruction from motion corrupted
stacks of 2D slices,” IEEE transactions on medical
imaging, vol. 34, no. 9, pp. 1901–1913, 2015.
[6] Sébastien Tourbier, Clemente Velasco-Annis, Vahid
Taimouri, Patric Hagmann, Reto Meuli, Simon K
Warfield, Cuadra M Bach, and Ali Gholipour, “Automated template-based brain localization and extraction
for fetal brain MRI reconstruction.,” NeuroImage, 2017.
[7] Kevin Keraudren,
Maria Kuklisova-Murgasova,
Vanessa Kyriakopoulou, Christina Malamateniou,
Mary A Rutherford, Bernhard Kainz, Joseph V Hajnal,
and Daniel Rueckert, “Automated fetal brain segmentation from 2d mri slices for motion correction,”
NeuroImage, vol. 101, pp. 633–643, 2014.
[8] Kevin Keraudren, Vanessa Kyriakopoulou, Mary
Rutherford, Joseph V Hajnal, and Daniel Rueckert,
“Localisation of the brain in fetal MRI using bundled
SIFT features,” in International Conference on Medical
Image Computing and Computer-Assisted Intervention.
Springer, 2013, pp. 582–589.
[9] Vahid Taimouri, Ali Gholipour, Clemente VelascoAnnis, Judy A Estroff, and Simon K Warfield, “A
template-to-slice block matching approach for automatic localization of brain in fetal MRI,” in Biomedical
Imaging (ISBI), 2015 IEEE 12th International Symposium on. IEEE, 2015, pp. 144–147.
[10] Sébastien Tourbier, Patric Hagmann, Maud Cagneaux,
Laurent Guibaud, Sai Subrahmanyam Gorthi, Marie

[11]

[12]

[13]

[14]

[15]

[16]

[17]

[18]

[19]

[20]

Schaer, Jean-Philippe Thiran, Reto Meuli, and Meritxell Bach Cuadra, “Automatic brain extraction in fetal
MRI using multi-atlas-based segmentation,” in Medical
Imaging: Image Processing, 2015, p. 94130Y.
N Khalili, P Moeskops, NHP Claessens, S Scherpenzeel, E Turk, R de Heus, MJNL Benders, MA Viergever,
JPW Pluim, and I Išgum, “Automatic segmentation of
the intracranial volume in fetal MR images,” in Fetal,
Infant and Ophthalmic Medical Image Analysis, pp. 42–
51. Springer, 2017.
Pim Moeskops, Max A Viergever, Adriënne M Mendrik, Linda S de Vries, Manon JNL Benders, and Ivana
Išgum, “Automatic segmentation of MR brain images
with a convolutional neural network,” IEEE transactions on medical imaging, vol. 35, no. 5, pp. 1252–1261,
2016.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox,
“U-net: Convolutional networks for biomedical image
segmentation,” in International Conference on Medical
Image Computing and Computer-Assisted Intervention.
Springer, 2015, pp. 234–241.
Seyed Sadegh Mohseni Salehi, Deniz Erdogmus, and
Ali Gholipour, “Auto-context convolutional neural network (auto-net) for brain extraction in magnetic resonance imaging,” IEEE Transactions on Medical Imaging, 2017.
Jonathan Long, Evan Shelhamer, and Trevor Darrell,
“Fully convolutional networks for semantic segmentation,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 3431–
3440.
Evan Shelhamer, Jonathan Long, and Trevor Darrell,
“Fully convolutional networks for semantic segmentation,” IEEE transactions on pattern analysis and machine intelligence, vol. 39, no. 4, pp. 640–651, 2017.
Diederik Kingma and Jimmy Ba,
“Adam: A
method for stochastic optimization,” arXiv preprint
arXiv:1412.6980, 2014.
Özgün Çiçek, Ahmed Abdulkadir, Soeren S Lienkamp,
Thomas Brox, and Olaf Ronneberger, “3D U-Net:
learning dense volumetric segmentation from sparse
annotation,” in International Conference on Medical
Image Computing and Computer-Assisted Intervention.
Springer, 2016, pp. 424–432.
Seyed Sadegh Mohseni Salehi, Deniz Erdogmus, and
Ali Gholipour, “Tversky loss function for image segmentation using 3D fully convolutional deep networks,”
in International Workshop on Machine Learning in
Medical Imaging. Springer, 2017, pp. 379–387.
P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y. LeCun, “Overfeat: Integrated recognition, localization and detection using convolutional networks,” arXiv preprint arXiv:1312.6229, 2013.

