Semantic Segmentation of Human Thigh Quadriceps
Muscle in Magnetic Resonance Images
E. Ahmad∗,1 , M. Goyal∗,1 , J. S. McPhee2 , H. Degens2 , and M. H. Yap1

arXiv:1801.00415v1 [] 1 Jan 2018

Manchester Metropolitan University, Manchester, United Kingdom

Abstract
This paper presents an end-to-end solution for MRI thigh quadriceps segmentation. This is the first attempt that deep learning methods are used for the MRI
thigh segmentation task. We use the state-of-the-art Fully Convolutional Networks with transfer learning approach for the semantic segmentation of regions
of interest in MRI thigh scans. To further improve the performance of the segmentation, we propose a post-processing technique using basic image processing
methods. With our proposed method, we have established a new benchmark
for MRI thigh quadriceps segmentation with mean Jaccard Similarity Index of
0.9502 and processing time of 0.117 second per image.
Key words: Semantic segmentation, Magnetic resonance imaging, Thigh
muscles, Deep learning, Fully convolutional network.

1. Introduction
There are over 600 skeletal muscles in the human body which play important
roles in: 1) thermoregulation by generating heat as a product of their contraction
and metabolism; 2) whole-body energy balance by utilizing and storing fatty
acids, glucose and amino acids consumed as part of the diet to provide energy for
movement and other cellular processes; and 3) paracrine and endocrine functions
as they release growth hormones and other factors into circulation [1]. However,
∗ Authors

Contributed Equally
of Computing, Mathematics and Digital Technology.
2 School of Healthcare Science.
1 School

Preprint submitted to Journal of Computerized Medical Imaging and Graphics. Dec 31, 2017

the most important functions of muscles are to provide support for posture and
to make movement possible by applying force while shortening to bones and
joints.
The muscles of the arms and legs principally allow us to interact with and
move around the environment. Of all the limb muscles, those located in the thigh
are amongst the largest and most powerful. The thigh is divided into three main
compartments: 1) the quadriceps muscle group on the anterior aspect; 2) the
adductor muscles that bring the thigh toward the mid-line and rotate it (on the
medial section); and 3) the hamstrings on the posterior aspect.
Skeletal muscles are highly adaptable to their habitual use: regular exercise
will induce muscles growth, while sedentary living or other types of disuse leads
to muscle atrophy. Indeed, any disease or other long-term condition causing
low muscle mass and strength can result in mobility impairments. The normal
ageing process is one such condition. During ageing, physical activity levels
decrease [2] and the consequent disuse alongside other biological processes such
as hormonal and/or other endocrine changes contribute to muscle wasting [3].
Another study has revealed that leg muscles of older people begin to accumulate fatty adipose tissue. There is very little information currently available
showing how adipose tissue accumulates in leg muscles. Quantification of the
changes in adipose tissue with ageing coupled with more detailed analysis of
muscle biopsy samples could help to identify the regulatory mechanisms leading
to loss of muscle mass and strength in older age because fatty adipose tissue
can emit proteins that promote the breakdown of surrounding tissue [4]. Hence,
The study of thigh muscles is a priority due to their importance in locomotion,
mobility and metabolism and the extent of their deterioration in ageing and
disease.
Artificial Intelligence is showing improvements in the autonomous and unsupervised learning systems, which in turn, revolutionize many industries and
bring significant shifts in society through developments in healthcare analysis
and treatment [5, 6, 7]. Technological advances in medical imaging over the past
three decades have led to greater accessibility to advanced imaging techniques
2

in clinical practice and research settings. Such technologies include magnetic
resonance imaging (MRI), dual-energy X-ray absorptiometry, ultrasonography
and computed tomography. They are widely used by clinicians to scan body
segments to diagnose injury or disease [8, 9]. These imaging techniques are
also used by researchers interested in studies of human anatomy and physiology
in health and disease across the lifespan. These techniques produce precise,
high-quality distinctions between different tissue and cell types.
In computer vision analysis and medical imaging, semantic segmentation is
an attempt to partition meaningful parts that belong to the same object class
of an image together and is mostly applied in object detection and localization
[10, 11]. In this work, five state-of-the-art semantic segmentation deep learning
architectures for regions of interest (ROI) in MRI of thigh muscles are reviewed.
The significant contributions of this work are:
1. To provide a review of current state-of-the-art image processing, traditional machine learning and deep learning approaches for segmentation of
MRI thigh muscles.
2. We present the largest dataset of MRI thigh muscles with annotated ROI
that consists of quadriceps, bone and marrow as ground truths from the
experts.
3. To date, this is the first time deep learning methods are used to segment
the ROI from the MRI of thigh muscles and results are compared with
the traditional methods.
4. We propose a post-processing technique which helps in further refinement
of segmentation of ROI in MRI of thigh muscles.

2. Related Work
From 2002 until 2015, twelve automated segmentation attempts on the MRI
thigh muscles have been recorded [12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23]),
where automation process was done by incorporating either one or combination of these techniques: thresholding (intensity based or histogram modelling);
3

classification (fuzzy c-means (FCM) being the popular approach or k-means);
active contour; and/or region growing. These methods focusing on segmenting muscles, marrow, femur, subcutaneous adipose tissue and/or intermuscular
adipose tissue as an individual component (or group), whereafter the implementation of suitable pre-processing algorithms (to remove/reduce noise or improve
pixel’s intensity), this individual component (or group) can be straightforwardly
segmented.
The segmentation accuracies across all methods above are considered exceptional in general, with all methods achieved average accuracy of greater than
85% and the segmentation results can be seen statistically improved over time
(2006/07 [14, 15] - 95.73%, by FCM; 2009 [17] - about 96%, by adaptive thresholding and histogram modelling; 2013 [22] - about 97%, by k -means clustering;
and 2015 [23] - 96.8%, by snake active contour). Considering the application
(or combination of applications) of such basic techniques, the system’s average
processing time across all methods are also recorded to be improved over time
(2009 [18] - 52 sec per image, by FCM and active contour; 2013 [21] - about 5.21
sec per image, by region growing and 3D intensity map; and 2015 [22] - 0.25
sec per image, by k -means clustering). Two major reasons that understandably
contribute to the better mean average processing time are superior computer
technology and hardware available on the market and the optimisation of algorithms and processing platforms from the developer. Regardless, this simple,
precise and fast automation process may also assist significantly in quantification analysis for cross-sectional area or volumetric of MRI of the thigh, to a
certain extent.
In 2012, an advanced discrete optimization solution by a graph-based Random Walker (RW) has been proposed by Baudin et al. [24], where a graphical
model that capable to automatically determine appropriate seed positions with
respect to different muscle classes has been introduced. The process was established by combining frameworks of seeds sampling and graph edges; and Markov
Random Fields formulation that calculate the cost function form, unary potential, geodesic distance potential, and relative orientation potential of muscle
4

structure and position. Authors reported that small muscles (within the thigh
region) are prone to segmentation errors compared to the large ones and as
a result, recommending the use of deformable registration approaches to alleviate the bias introduced from the rigid registration step, for future work.
Next, in [25], the same authors worked on the introduction of linear sub-spaces
constraints within RW segmentation framework, where the novel knowledge or
prior-based quadriceps segmentation was proposed, contributing to a slightly improved in segmentation output accuracy, compared to [24]. The technique was
developed based on RW formulation, where the concatenation of all the node’s
(or seed’s) probability is proliferated by block diagonal matrix (depending on
the quantity of the label); combined with prior knowledge to the respective RW
formulation above; and the design of low dimensional affine of implicit space
within the spatial matrix. Results were presented in 2D MRI cross-sectional
area, with an average segmentation accuracy (in Dice Similarity Coefficient
(DSC)) of 0.84 ± 0.08 and computational time of approximately 15 minutes
per segmentation. Another effort from Baudin et al. including an extension
for the segmentation of whole muscles of thigh MRI [26], again by integration
of similar RW graph-based with prior information frameworks as above, where
statistical shape atlas was employed to represent prior knowledge. Apart from
this frameworks, a balancing parameter was introduced to the formulation of
nodal’s total energy; with an addition of confidence map to adjust the influence
of contour’s model. This configuration leads to an efficient iterative linear optimization and further enhanced the quality of segmentation accuracy, yielding
an average DSC of 0.86 ± 0.07 and processing time of around 5 minutes per segmentation. However, errors in muscle segmentation are still occurred, especially
to small muscle components (e.g. Gracilis muscle and tensor fasciae latae), due
to large registration errors on the same muscle assumption.
Another fully automatic segmentation method for thigh MRI was demonstrated by Andrews et al. [27]. This technique provides a good accuracy for individual muscle segmentation of the thigh, by using energy minimizing probabilistic segmentation that indicates area of ambiguity. The automatic segmentation
5

made possible with the frameworks that comprise of a probabilistic segmentation function representation of isometric log-ratio (ILR) transform; the shape
space (relative locations of muscles) by principal component analysis (PCA);
image alignment by using femur bone coordination as an anchor of the vector
estimation; and energy construction strictly based on energy function over the
shape space of an image. The results were presented in 3D and the method
capable to segment 11 muscles within the thigh automatically and achieved a
mean DSC of 0.92 ± 0.03 across all images, but with undisclosed processing
time per segmentation/image. A recent publication by Andrews and Hamarneh
[28] presents comprehensive incorporation of probabilistic shape representation
consists of adjacent object information and prior anatomical volume, called generalized log-ratio (GLR). The main benefit of GLR application is that it may
produce a probabilistic segmentation that can be used to generate uncertainty
information to aid subsequent analysis of different muscle components on thigh
MRI. In other words, the GLR transformation can be designed to ensure statistical shape models capture variability in smaller structures properly. The
method is designed based on four major steps. Firstly, GLR representation in
the context of thigh muscle segmentation to encode details such as muscle size,
adjacency information, and to train a statistical shape model over the space
of GLR representations. Secondly, the introduction of pre-segmenting images
technique for all components involve (fat, muscles and bone). The results are
mainly used to perform alignment computation. Thirdly, random forest classifier training to assist in detecting intermuscular boundary locations. And finally,
results from all steps above were integrated into a globally minimized convex
energy functional by the mean of primal-dual method to generate a probabilistic
segmentation. As the system was mainly designed to tackle issues related to
segmentation accuracy, the major drawback is that the average segmentation
time took 50 ± 4.3 minutes per image to run. The mean DSC by using this
method is 0.808 ± 0.074, which is quite promising, considering the procedure
and analysis were done to all muscles, individually, on MRI of the thigh.
In 1992, Morrison et al. [29] made an attempt on MRI thigh segmentation
6

by using neural network. The network model integrates a probabilistic neural
network to facilitate the generation of probability estimates at each pixel for
use in an iterative segmentation process. This preliminary study on MRI thigh
segmentation by using one of the simplest forms of neural network reported
that the segmentation result is unaffected by the quality of the training dataset.
Today, the concept of neural network has been revolutionised and diversely
applied, including in segmentation of medical imaging.
Recently, deep learning has provided various end-to-end solutions for the
semantic segmentation of abnormalities such as breast cancer, brain tumour,
skin lesions, foot ulcers in various image modalities of medical imaging [30,
31, 32]. At the time of writing, there is no other study has been made in the
segmentation of MR images of human thigh muscles with deep learning methods.
Current state-of-the-art methods, especially image processing based techniques,
are not robust, due to their nature of reliance on specific regulators and rules,
with certain assumptions. In contrast to conventional machine learning, deep
learning methods do not require such strong assumptions and have demonstrated
superiority in region and object segmentation and classification, which suggests
that the robust fully automated MRI segmentation of human thigh muscles may
be achieved, by adopting such approach. Moreover, the segmentation output
performance from most recent applications of network models in MRI of different
domains [33, 34, 35], in association with image processing technique [36] and in
conjunction with the utilization of better and upgradable graphics processing
unit (GPU), is remarkably accurate and fast.

3. Experiments
This section discusses the selection and labelling of datasets of MRI of thigh
muscles, the preparation of the ground truth image and object labelling and the
proposed methods for experimentation of semantic segmentation.

7

3.1. MRI Datasets of Thigh Muscles
All MRI scans of the thigh were collected from men and women aged 18-90 years
[37] by using the same T1-weighted Turbo 3D sequences from a 0.2-Tesla MRI
scanner (Esaote G-scan; Italy). All scans consisted of serial transverse-plane
slices: 1) each with 6.3-mm thickness and 0-mm inter-slice gap; 2) image matrix
of 256 x 256 (in pixels); and 3) assembled in either 13 or 26 sequential scans per
individual subject. Figures 1 and 2 demonstrate both standard types of scan
(per individual subject) with 13 and 26 scans, respectively.

Figure 1: 13 sequential scans per individual subject.

In this work, we created three types of MRI of the thigh datasets. The
first dataset is labelled as mid-scan (MD), where a single middle-scanned MRI
is selected from each dataset of 110 different subjects, i.e. MRI scan number
7 of individual subject with 13 scans (Figure 1) and MRI scan number 13 of
individual subject with 26 scans (Figure 2). The second dataset is labelled as
whole-scan (WD), which is the combination of all scans from both 13 and 26
scans datasets of 50 individual subjects to form a WD dataset with 1000 MR
images. The third dataset is labelled as all-scan (AD), which is a combination
of both MD and WD datasets.
8

Figure 2: 26 sequential scans per individual subject

Since, most of the pre-trained deep learning frameworks were trained on the
three channel images such as RGB format, all of the original 1000 grayscale
MRI are converted to the images of 3-channel colour profile of sRGB IEC619662.1. The binary ground truth masks which are delineated by domain experts
to represent ROI are converted into the Pascal VOC format (labelled as 8-bit
single-channel paletted images) [11].
In this Pascal VOC format, 0 maps are indexed to black pixels corresponding
to the background and 1 (in red) denotes the ROI (consists of quadriceps, femur
and marrow). This procedure is illustrated in Figure 3 below.
3.2. Deep Learning Architectures for Semantic Segmentation
This section provides the brief description of deep learning architectures used
for the semantic segmentation of ROI in the MRI of thigh muscles.
FCN-AlexNet. FCN-AlexNet is a customized FCN model based on AlexNet
architecture [38]. Since AlexNet specializes in object classifications (won 1st
position for ImageNet competition in 2012 with dataset containing 15 million of
images and with 1000 of different object classes), a few adjustments are made

9

Figure 3: Preparation of MRI Datasets.

in the network layers to make architecture capable of doing the segmentation
task. Earlier, network’s convolution layers are maintained for as usual features
extraction, but, the fully connected (FC) layers are removed and replaced with
equivalent convolution layers as FC layers throw away the spatial information
which is crucial for the segmentation task.
FCN-VGGNet. In 2014, K. Simonyan and A. Zisserman of the Visual Geometry Group (VGG) from the University of Oxford proposed a CNN model
known as VGG-16 [39] and secured the first and second place in localization
and classification, respectively in the ImageNet Challenge 2014, with a test error
rate of 7.3% and test accuracy of 92.7% in ILSVRC 2014 submission. VGG-16
10

architecture is customized to three models, FCN-32s, FCN-16s and FCN-8s,
where each model magnifies the output with different upsampling layers [11].
For FCN-32s, the fully connected layers are convolutionized and a 32-pixel stride
size is applied in deconvolution layers. This works well for object classification,
localization and detection tasks as precise pixel-wise predictions are not the
main priority. FCN-16s and FCN-8s perform better for object segmentation
tasks, as both extract extra low-level features from the input image to produce
a more precise output result by 16 x 16 and 8 x 8 pixel blocks, respectively.
In FCN-16s, the final output is a product of upsampling of two layers, which
are upsampled from the fourth pooling layer and (upsampling of the seventh
convolutional layer) x 2. For FCN-8s, the final output of the model is a product
of upsampling of the third pooling layer, (upsampling of the fourth pooling
layer) x 2 and (upsampling of the seventh convolutional layer) x 4.
PSPNet. In 2016, Zhou et al. proposed a network architecture called Pyramid Scene Parsing Network (PSPNet) [40]. In this work, the research team from
Chinese University of Hong Kong and SenseTime Group Limited used the global
context information proficiency by aggregation of a different-region-based context. The global spatial context is important since it provides suggestions on the
distribution of the segmentation classes and therefore, tackling issues related to
mismatched relationship, confusion categories and inconspicuous classes. This
procedure is done by the combination of a pyramid pooling module (applying
four different sizes of kernel pooling layers) and proved to have an effective
global prior representation capable to produce good results in the scene parsing
tasks and pixel-level predictions. Other key features of this architecture include:
integrating dilated convolutions as a means of modifying the base architecture
of Residual Network ResNet [41], a 152-layer network architecture proposed by
Microsoft Research Asia in late 2015. The proposed method achieved the first
place in ImageNet scene parsing challenge 2016, PASCAL VOC 2012 benchmark
and Cityscapes benchmark. The paper also reported that PSPNet model yields
an accuracy of 85.4% (by mean intersection over union (mIoU)) on PASCAL

11

VOC 2012 and accuracy of 80.2% on Cityscapes.
3.2.1. Transfer Learning
Having a sufficient amount of data is crucial to effectively train a deep neural
network. However, in reality especially for medical imaging, it is not often
possible to have large datasets and this is why the idea of transfer learning
has emerged in recent years to produce significant results for smaller medical
imaging datasets. Instead of training the deep learning methods from scratch,
transfer learning is a method that uses the weights from pre-trained models and
then, fine-tuning the network with the new dataset.
In this work, the weights associated to these pre-trained models are subsequently used to fine-tune the deep learning methods for training the thigh MRI
datasets. This transfer learning process is demonstrated in Figure 4.
Configuration of GPU Machine for Experiments. (1) Hardware: CPU
- Intel i7-6700 @ 4.00Ghz, GPU - NVIDIA 1080 TITAN X 11Gb, RAM - 32Gb
DDR5 (2) Software: Caffe [42].

4. Results and Discussion
Two datasets including a total of 1110 MR images were manually labelled by the
domain experts. These datasets of MRI are then trained, validated and tested by
the state-of-the-art semantic segmentation models that are FCN-AlexNet, FCNVGG-16 (consists of FCN-32s, FCN-16s and FCN-8s) and PSPNet. Both midscan (MD) and whole-scan (WD) MRI datasets were split into the configuration
of 70%, 10% and 20% for training, validation and testing, respectively (MD:
77, 11, 22 images and WD: 700, 100, 200 images), with 5-fold cross-validation
procedure (5 processing batches to analyse all MR images). These models were
trained on deep learning library called Caffe [42] with solver type is stochastic
gradient descent [43], the learning rate of 0.0001 and number of epochs are set
to 50.

12

Figure 4: Transfer learning procedure of deep CNNs to obtain optimized weights initializations. Three fully connected layers of CNN were removed and replaced by three convolutional
layers, making the pre-trained model fully convolutional.

To evaluate the performance of these FCN models, all of the semantically
segmented (tested) images were compared and measured with the corresponding
labelled (manually segmented ROI) images. In addition to Jaccard Similarity
Index (JSI), Sensitivity, Specificity, Matthews Correlation Coefficient (MCC)
and Dice Similarity Coefficient (DSC) indexes were also employed as the performance metrics for the evaluation of segmented ROI. MCC [44] is defined as:

13

TP × TN − FP × FN
,
M CC = p
(T P + F P )(T P + F N )(T N + F P )(T N + F N )

(1)

where TP, TN, FP and FN denote True Positives, True Negatives, False Positives and False Negatives, respectively.
Table 1 summarizes the mean index of semantic segmentation results of all
the deep learning models on mid-scan (MD) and whole-scan (WD) datasets.
Table 1: Mean performance index for all models on mid-scan (MD) and whole-scan (WD)
datasets.
JSI

DSC

Sensitivity

Specificity

MCC

Network

MD

WD

MD

WD

MD

WD

MD

WD

MD

WD

PSPNet

0.8263

0.9194

0.9026

0.9565

0.9029

0.9583

0.9252

0.9611

0.828

0.9209

FCN-AlexNet

0.8487

0.8923

0.9169

0.9332

0.901

0.9446

0.9804

0.9769

0.8925

0.9153

FCN-32s

0.8198

0.9211

0.899

0.9581

0.8796

0.9768

0.9776

0.9467

0.8712

0.9232

FCN-16s

0.8669

0.9199

0.9266

0.9574

0.9399

0.9782

0.9743

0.9442

0.9059

0.922

FCN-8s

0.8973

0.9316

0.9442

0.9639

0.9708

0.9822

0.9748

0.9514

0.9282

0.9336

Overall, FCN-8s performed best in regards to the semantic segmentation task
of the ROI of thigh MR images, with an average accuracy of 0.8973 (89.73%) for
MD and 0.9316 (93.16%) for WD by JSI. The average DSC performance index
for the corresponding datasets was also notably higher than JSI, i.e. 0.9442
(94.42%) and 0.9639 (96.39%), for MD and WD, respectively. Due to the lessforgiving nature of its measurement, JSI will be used comprehensively from
here after. DSC was included partially because it was universally employed as
a means of performance evaluation in machine learning. Although both JSI and
DSC adapted a slightly different formulation from one another, both indexes
are good for performance assessment of segmentation task.
FCN-8s excelled throughout the performance indices used as it was ranked
first in JSI, DSC, Sensitivity and MCC and ranked third in the Specificity
performance index. However, the difference in Specificity was not substantial
from the first-ranked FCN-AlexNet model (differences of 0.0056 and 0.0255 for
MD and WD, respectively).

14

4.1. FCN-8s model with different solver types for MD and WD
Stochastic Gradient Descent (SGD) [43] is arguably predominantly used to train
deep learning models, mainly due to its strength of simplicity in implementation
and fast processing, even for problems (or datasets) with many training patterns.
Semantic segmentation results in Table 1 also generated by the application of
SGD as its solver type in all pre-trained models.
Table 2 demonstrates the performance impact on the application of different
solver methods used for semantic segmentation in thigh MR datasets by FCN-8s
model.
Table 2: Mean JSI performance index for different solver types used in FCN-8s on mid-scan
(MD) and whole-scan (WD) datasets.

MD

JSI

WD

AdaGrad [45]

Adam [46]

NAG [47]

RMSprop [48]

SGD

Adam

SGD

0.8778

0.927

0.8972

0.8829

0.8973

0.9469

0.9316

From Table 2, the various applications of solver types on FCN-8s during the
training and validation suggested that Adam solver generally performs better
for the semantic segmentation of ROI in thigh MRI, with both MD and WD
datasets having a JSI performance increment of 0.0297 (2.97%) and 0.0153
(1.53%) in terms of accuracy, respectively compared to FCN-8s with SGD solver.
Unfortunately, none of the results can be generated with AdaDelta solver [49]
on FCN-8s due to the lower percentage (≈ 70%) of learning accuracy and higher
loss during the training, even after the number of epoch was increased.
4.2. Segmentation of MRI thigh muscle on All-scan Dataset
The main intention of distributing the dataset into two (MD and WD) is to
observe for any significance in JSI performance index for the semantic segmentation task of the ROI of the thigh MRI due to the substantial differences of
the total number of images in these datasets (110 and 1000 images for MD and
WD, respectively). As it was proven by the results in the previous sections, the

15

performance of each model is marginally improved with the increased number
of images in the WD dataset.
We further investigate the performance of these models by combining the
MD and WD to form an all-scan dataset (AD). For this purpose, 1000 images
of WD are used for training and validation, whereas 110 images of MD are used
for testing. The new training, validation and testing configurations for AD are
then segregated as 900, 100 and 110 images, respectively. The hypothesis of
forming this dataset is that the testing performances for each model should be
improved as more images are incorporated in training the models, compared to
MD and WD datasets. Table 3 shows the results of this procedure, with results
of both MD and WD are also included, for comparisons.
Table 3: Mean JSI performance index of all models on mid-scan (MD), whole-scan (WD) and
all-scan (AD) datasets.

Model
FCN-AlexNet

Dataset

FCN-8s

FCN-16s

FCN-32s

PSPNet

Adam

SGD

Adam

SGD

SGD

SGD

SGD

MD

0.897

0.8487

0.927

0.8973

0.8669

0.8198

0.8263

WD

0.9253

0.8923

0.9469

0.9316

0.9199

0.9211

0.9194

AD

0.9228

0.8988

0.9388

0.9391

0.9378

0.9412

0.9081

As expected, overall FCN-8s (with Adam solver) performed better compared to the other models, throughout all the datasets (MD, WD or AD), with
the performance of FCN-8s for WD dataset (94.69%) remains the best. However, surprisingly, the incorporation of SGD solver on FCN-32s performed better
(compared to the other models within AD), with average JSI performance of
0.9412 (94.12%), an increment of 0.201 (or 2.01%) by JSI compared to FCN32s for WD. Also note that the mean JSI performances for FCN-8s (with both
Adam and SGD solver methods), FCN-16s and FCN-32s, in AD dataset are also
close to one another, with only a small performance difference between them
(±0.0034 or 0.34%). This suggests that earlier hypothesis is inaccurate for the
most of the deep learning models employed, as overall performance improved
16

only for FCN-32s. The main reason for failed hypothesis is more divergent testing data that consists of single scan of 110 individual subjects for training data
that consists of sequential scans of only 50 individual subjects in this dataset.
The average processing time to inference a single MRI of thigh muscle also
stands as an important performance indication for the means of evaluation of all
models. Table 4 depicts the average processing (semantic segmentation testing)
time (in seconds) for a single thigh MR image for all proposed models.
Table 4: Mean processing (testing) time (in seconds) for all models to semantically segment
an ROI of thigh MR image.
Model
FCN-AlexNet

Dataset

FCN-8s

FCN-16s

FCN-32s

PSPNet

Adam

SGD

AdaGrad

Adam

NAG

RMSprop

SGD

Adam

SGD

Adam

SGD

SGD

MD

0.34

0.24

0.46

0.64

0.45

0.37

0.61

0.58

0.39

0.61

0.55

0.05

WD

0.18

0.15

-

0.1

-

-

0.03

-

0.03

-

0.03

0.05

AD

0.1

0.09

-

0.17

-

-

0.03

-

0.04

-

0.03

0.05

FCN-8s with SGD solver (in AD and WD) was the best in terms of computation time for the semantic segmentation task of thigh MRI, with 30 ms (0.03
sec) average processing time per image. That was 100 times faster than the
proposed method in [37] (3 sec), 567 times faster than the proposed method in
[50] (17 sec) and 733 times faster the proposed method in [51]. In correlation to
the best performer in terms of accuracy (FCN-8s with Adam solver in WD; average processing time of 0.1 sec per image), the processing time was 30, 170 and
220 times faster than the frameworks in [37], [50] and [51], respectively. This
indicates that in general, the FCN-8s model is ideal for the domain of semantic
segmentation as it is not only accurate, but also considerably fast. Overall, the
application of any pre-trained model for testing (semantic segmentation) the
ROI of thigh MRI is reliable in terms of processing time, with FCN-8s (with
Adam solver in MD) taking the longest time - 0.64 sec. And that still is 4.7
times faster than the proposed method in [37].
This was also confirmed: 1) the general influence of utilizing the simplicity of
update value formulation of the weight Wt+1 [43] for SGD, evidently results in

17

faster computation, compared to the complexity of Adams’s weight update [46];
and 2) vice-versa for the accuracy department, as Adam generally performed
better throughout all datasets, compared to SGD.
4.3. Post-processing
Notwithstanding the promising results set out above, there is still a need to
investigate the possibility of improving the end results of semantic segmentation
by the FCN-8s model. Here, the inclusion of a post-processing stage by image
processing techniques will be explored, with the goal of enhancing the ROI
segmentation result. This FCN-8s model dedicated experimentations will be
applied to both Adam and SGD solvers on all datasets (MD, WD and AD). The
process involves image filtering and morphological procedures to the segmented
ROI. Figure 5 demonstrates the significance of the inclusion of this stage and
Table 5 shows the mean JSI performance index with the implementation of
post-processing stage by image processing techniques described earlier.

Figure 5: Image filtering and morphological to the semantic segmentation image. (a) Ground
truth label (in white) with original MR image. (b) Semantic segmentation result by FCN8-s
model (for this sample: with Adam solver in WD dataset). (c) Post-processing result of (b).

The results above demonstrated that by incorporating the proposed postprocessing stage, the semantic segmentation output can be enhanced. The overall accuracy performances are increased throughout all of the datasets, with
FCN-8s (Adam solver in WD) achieved the best result. However, the performance increment was minimal, due to the fact that only < 1% (or 0.86% to be
exact across output in Table 5) of semantically segmented images are affected
18

Table 5: Mean JSI performance index of FCN-8s model on mid-scan (MD), whole-scan (WD)
and all-scan (AD) datasets, without and with post-processing, respectively.

FCN-8s

Dataset

Without post-processing

With post-processing

Adam

SGD

Adam

SGD

MD

0.927

0.8973

0.927

0.9006

WD

0.9469

0.9316

0.9502

0.9325

AD

0.9388

0.9391

0.9398

0.9392

by this over-segmentation condition as shown in Figure 5(b). With additional
processing, the overall mean processing time to process a thigh MR image also
increased. It is measured that an extra 17 ms (0.017 sec) is needed for this purpose (for FCN-8s with Adam solver in WD), increasing the overall processing
time to 0.1 + 0.017 = 0.117 sec.
4.4. Comparison with benchmark algorithms
The closest state-of-the-art segmentation methods using the similar definition of ROIs were done by Ahmad et. al. [50, 37, 51]. They first established
a semi-automated segmentation method for mid-scan ROI segmentation [50]
and achieved a mean JSI of 0.95 (or 95%). The work was further investigated
towards the automatic segmentation of sequential MR images in a dataset (of
a single subject) [37] by the combination frameworks of the earlier method in
[50] and image processing techniques and achieved an average JSI of 0.9334 (or
93.34%). Although both frameworks yield reliable and accurate segmentation
results, these frameworks still need minimal supervision to precisely delineate
the muscle borders. A fully automated segmentation method was later developed in [51] by the integration of statistical analysis, image processing and
computer vision algorithms, which requires no interactions from the user and
offers good performance in terms of segmentation accuracy (mean JSI of 0.8526)
of the ROI (quadriceps, femur and marrow) and processing time.

19

This paper has demonstrated that FCN-8s with post-processing has outperformed the benchmark algorithms and therefore warrants the in-depth analysis before further application. In addition, in terms of processing time, it was
26, 145 and 188 times faster than the frameworks in [37], [50] and [51], respectively.

5. Conclusion
In this work, five deep learning models FCN-AlexNet, FCN-32s, FCN-16s, FCN8s and PSPNet have been explored for the semantic segmentation task of the
ROI which include quadriceps, bone, and marrow on MRI of thigh muscles.
FCN-8s emerges as the best all-around deep learning model for the task, and
Adam solver type works the best overall for the thigh MR images, as compared
to the other solver types. The FCN-8s with Adam solver, trained, validated and
tested on mid-scan (MD), whole-scan (WD) and all-scan (AD) datasets, produced a mean semantic segmentation accuracy by JSI of 0.927 (92.7%), 0.9469
(94.69%) and 0.9388 (93.88%), within an average processing time per image of
0.64 sec, 0.1 sec and 0.17 sec, respectively for the corresponding datasets.
With the implementation of post-processing stage of image filtering and morphology to the ROI segmented output, the results are enhanced to 0.927 (92.7%),
0.9502 (95.02%) and 0.9398 (93.98%) for MD, WD and AD, respectively. The
best JSI performance result of 0.9502 (95.02%) suggests that FCN-8s model
with Adam solver in WD has out-performed the mean JSI benchmark of 0.95
(or 95%) set in [50] for the semi-automatic segmentation framework. However,
this semi-automatic segmentation framework suggested in [50] requires manual
tuning of image processing parameters for the individual cases which can be
very tedious work especially for domain experts. Whereas, the deep learning
models present end-to-end solution with quick processing time for inferencing.
The output performance of FCN-8s (without the inclusion of post-processing
stage) i.e. 0.947 (or 94.7 %) is comparable to the previous benchmark achieved.
In the future work: 1) to explore other potential deep learning architec-

20

tures that are used for the semantic segmentation task (such as U-Net [52],
SegNet [53] and FCN-DenseNet [54]) and investigate, evaluate and compare
their performances with the current benchmark method; and 2) the prospect of
developing novel deep learning architecture that specifies and optimizes to the
ROI segmentation with post-processing technique included in it.

References
[1] P. Blakey, T. E. Educational, The muscle book, Bibliotek Books, 1992.
[2] J. F. Sallis, Age-related decline in physical activity: a synthesis of human
and animal studies., Medicine and science in sports and exercise 32 (9)
(2000) 1598–1600.
[3] H. Degens, The role of systemic inflammation in age-related muscle weakness and wasting, Scandinavian journal of medicine & science in sports
20 (1) (2010) 28–38.
[4] F. Picard, L. Guarente, Molecular links between aging and adipose tissue,
International Journal of Obesity 29 (2005) S36–S39.
[5] Z. Liang, G. Zhang, J. X. Huang, Q. V. Hu, Deep learning for healthcare
decision making with emrs, in: Bioinformatics and Biomedicine (BIBM),
2014 IEEE International Conference on, IEEE, 2014, pp. 556–559.
[6] S. R. Granter, A. H. Beck, D. J. Papke Jr, Straw men, deep learning, and
the future of the human microscopist: Response to artificial intelligence and
the pathologist: Future frenemies?, Archives of Pathology & Laboratory
Medicine 141 (5) (2017) 624–624.
[7] M. Goyal, N. D. Reeves, A. K. Davison, S. Rajbhandari, J. Spragg, M. H.
Yap, Dfunet: Convolutional neural networks for diabetic foot ulcer classification, arXiv preprint arXiv:1711.10448.

21

[8] M. H. Yap, E. Edirisinghe, H. Bez, Processed images in human perception:
A case study in ultrasound breast imaging, European journal of radiology
73 (3) (2010) 682–687.
[9] M. H. Yap, A. G. Gale, H. J. Scott, Generic infrastructure for medical
informatics (gimi): the development of a mammographic training system,
in: International Workshop on Digital Mammography, Springer, 2008, pp.
577–584.
[10] M. Thoma,

A survey of semantic segmentation,

arXiv preprint

arXiv:1602.06541.
[11] E. Shelhamer, J. Long, T. Darrell, Fully convolutional networks for semantic segmentation, IEEE transactions on pattern analysis and machine
intelligence 39 (4) (2017) 640–651.
[12] V. Barra, J.-Y. Boire, Segmentation of fat and muscle from mr images of
the thigh by a possibilistic clustering algorithm, Computer methods and
programs in biomedicine 68 (3) (2002) 185–193.
[13] J. Mattei, Y. L. Fur, N. Cuge, S. Guis, P. Cozzone, D. Bendahan, Segmentation of fascias, fat and muscle from magnetic resonance images in
humans: the dispimag software, Magnetic Resonance Materials in Physics,
Biology and Medicine 19 (5) (2006) 275–279.
[14] H. Kang, A. Pinti, L. Vermeiren, A. Taleb-Ahmed, X. Zeng, An automatic
fcm-based method for tissue classification application to mri of thigh, in:
Bioinformatics and Biomedical Engineering, 2007. ICBBE 2007. The 1st
International Conference on, IEEE, 2007, pp. 510–514.
[15] H. Kang, A. Pinti, L. Vermeiren, A. Taleb-Ahmed, X. Zeng, Tissue classification for mri of thigh using a modified fcm method, in: Engineering in
Medicine and Biology Society, 2007. EMBS 2007. 29th Annual International
Conference of the IEEE, IEEE, 2007, pp. 5579–5584.

22

[16] C. Wang, X. Guo, L. Yao, K. Li, Z. Jin, A practical method for muscles
extraction and automatic segmentation of leg magnetic resonance images,
in: Complex Medical Engineering, 2007. CME 2007. IEEE/ICME International Conference on, IEEE, 2007, pp. 885–890.
[17] L. Urricelqui, A. Malanda, A. Villanueva, Automatic segmentation of thigh
magnetic resonance images, World Academy of Science, Engineering and
Technology 34.
[18] V. Positano, T. Christiansen, M. F. Santarelli, S. Ringgaard, L. Landini,
A. Gastaldelli, Accurate segmentation of subcutaneous and intermuscular
adipose tissue from mr images of the thigh, Journal of Magnetic Resonance
Imaging 29 (3) (2009) 677–684.
[19] P. Jiang, C. Yang, S. Wu, X. Ruan, The research of tissue segmentation
method on thigh mri, in: Multimedia Technology (ICMT), 2010 International Conference on, IEEE, 2010, pp. 1–3.
[20] S. Makrogiannis, S. Serai, K. W. Fishbein, C. Schreiber, L. Ferrucci, R. G.
Spencer, Automated quantification of muscle and fat in the thigh from
water-, fat-, and nonsuppressed mr images, Journal of Magnetic Resonance
Imaging 35 (5) (2012) 1152–1161.
[21] S. Purushwalkam, B. Li, Q. Meng, J. McPhee, Automatic segmentation of
adipose tissue from thigh magnetic resonance images, in: Image Analysis
and Recognition, Springer, 2013, pp. 451–458.
[22] A. Valentinitsch, D. C Karampinos, H. Alizai, K. Subburaj, D. Kumar,
T. M Link, S. Majumdar, Automated unsupervised multi-parametric classification of adipose tissue depots in skeletal muscle, Journal of Magnetic
Resonance Imaging 37 (4) (2013) 917–927.
[23] S. Orgiu, C. L. Lafortuna, F. Rastelli, M. Cadioli, A. Falini, G. Rizzo,
Automatic muscle and fat segmentation in the thigh from t1-weighted mri,
Journal of Magnetic Resonance Imaging.
23

[24] P.-Y. Baudin, N. Azzabou, P. G. Carlier, N. Paragios, Automatic skeletal
muscle segmentation through random walks and graph-based seed placement, in: Biomedical Imaging (ISBI), 2012 9th IEEE International Symposium on, IEEE, 2012, pp. 1036–1039.
[25] P.-Y. Baudin, N. Azzabou, P. G. Carlier, N. Paragios, et al., Manifoldenhanced segmentation through random walks on linear subspace priors,
in: Proceedings of the British Machine Vision Conference, 2012.
[26] P.-Y. Baudin, N. Azzabou, P. G. Carlier, N. Paragios, Prior knowledge,
random walks and human skeletal muscle segmentation, in: Medical Image
Computing and Computer-Assisted Intervention–MICCAI 2012, Springer,
2012, pp. 569–576.
[27] S. Andrews, G. Hamarneh, A. Yazdanpanah, B. HajGhanbari, W. D. Reid,
Probabilistic multi-shape segmentation of knee extensor and flexor muscles, in: Medical Image Computing and Computer-Assisted Intervention–
MICCAI 2011, Springer, 2011, pp. 651–658.
[28] S. Andrews, G. Hamarneh, The generalized log-ratio transformation:
Learning shape and adjacency priors for simultaneous thigh muscle segmentation.
[29] M. Morrison, Y. Attikiouzel, A probabilistic neural network based image
segmentation network for magnetic resonance images, in: Neural Networks,
1992. IJCNN., International Joint Conference on, Vol. 3, IEEE, 1992, pp.
60–65.
[30] M. Goyal, N. D. Reeves, S. Rajbhandari, J. Spragg, M. H. Yap, Fully
convolutional networks for diabetic foot ulcer segmentation, arXiv preprint
arXiv:1708.01928.
[31] M. H. Yap, G. Pons, J. Martı́, S. Ganau, M. Sentı́s, R. Zwiggelaar, A. K.
Davison, R. Martı́, Automated breast ultrasound lesions detection using

24

convolutional neural networks, IEEE Journal of Biomedical and Health
Informatics.
[32] M. Goyal, M. H. Yap, Multi-class semantic segmentation of skin lesions via
fully convolutional networks, arXiv preprint arXiv:1711.10449.
[33] S. Liao, Y. Gao, A. Oto, D. Shen, Representation learning: a unified deep
learning framework for automatic prostate mr segmentation, in: International Conference on Medical Image Computing and Computer-Assisted
Intervention, Springer, 2013, pp. 254–261.
[34] C. F. Baumgartner, L. M. Koch, M. Pollefeys, E. Konukoglu, An exploration of 2d and 3d deep learning techniques for cardiac mr image segmentation, arXiv preprint arXiv:1709.04496.
[35] S. Trebeschi, J. J. van Griethuysen, D. M. Lambregts, M. J. Lahaye,
C. Parmer, F. C. Bakers, N. H. Peters, R. G. Beets-Tan, H. J. Aerts,
Deep learning for fully-automated localization and segmentation of rectal
cancer on multiparametric mr, Scientific Reports 7 (1) (2017) 5301.
[36] M. Avendi, A. Kheradvar, H. Jafarkhani, A combined deep-learning and
deformable-model approach to fully automatic segmentation of the left ventricle in cardiac mri, Medical image analysis 30 (2016) 108–119.
[37] E. Ahmad, M. H. Yap, H. Degens, J. McPhee, Enhancement of mri human thigh muscle segmentation by template-based framework, in: Control
System, Computing and Engineering (ICCSCE), 2014 IEEE International
Conference on, IEEE, 2014, pp. 405–410.
[38] A. Krizhevsky, I. Sutskever, G. E. Hinton, Imagenet classification with
deep convolutional neural networks, in: Advances in neural information
processing systems, 2012, pp. 1097–1105.
[39] K. Simonyan, A. Zisserman, Very deep convolutional networks for largescale image recognition, arXiv preprint arXiv:1409.1556.

25

[40] H. Zhao, J. Shi, X. Qi, X. Wang, J. Jia, Pyramid scene parsing network,
arXiv preprint arXiv:1612.01105.
[41] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and
pattern recognition, 2016, pp. 770–778.
[42] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick,
S. Guadarrama, T. Darrell, Caffe: Convolutional architecture for fast feature embedding, in: Proceedings of the 22nd ACM international conference
on Multimedia, ACM, 2014, pp. 675–678.
[43] L. Bottou, Stochastic gradient descent tricks, in: Neural networks: Tricks
of the trade, Springer, 2012, pp. 421–436.
[44] B. W. Matthews, Comparison of the predicted and observed secondary
structure of t4 phage lysozyme, Biochimica et Biophysica Acta (BBA)Protein Structure 405 (2) (1975) 442–451.
[45] J. Duchi, E. Hazan, Y. Singer, Adaptive subgradient methods for online
learning and stochastic optimization, Journal of Machine Learning Research 12 (Jul) (2011) 2121–2159.
[46] D. Kingma, J. Ba, Adam: A method for stochastic optimization, arXiv
preprint arXiv:1412.6980.
[47] Y. Nesterov, A method of solving a convex programming problem with
convergence rate o (1/k2), in: Soviet Mathematics Doklady, Vol. 27, 1983,
pp. 372–376.
[48] G. Hinton, N. Srivastava, K. Swersky, Rmsprop: Divide the gradient by
a running average of its recent magnitude, Neural networks for machine
learning, Coursera lecture 6e.
[49] M. D. Zeiler, Adadelta: an adaptive learning rate method, arXiv preprint
arXiv:1212.5701.
26

[50] E. Ahmad, M. H. Yap, H. Degens, J. S. McPhee, Atlas-registration based
image segmentation of mri human thigh muscles in 3d space, in: SPIE
Medical Imaging, International Society for Optics and Photonics, 2014,
pp. 90371L–90371L.
[51] E. Ahmad, H. Degens, J. McPhee, M. H. Yap, Automatic segmentation
of mri human thigh muscles, in: Biomedical Engineering and Technology
(ICBET), 2018 International Conference on, ICBET, 2018.
[52] O. Ronneberger, P. Fischer, T. Brox, U-net: Convolutional networks for
biomedical image segmentation, in: International Conference on Medical
Image Computing and Computer-Assisted Intervention, Springer, 2015, pp.
234–241.
[53] V. Badrinarayanan, A. Kendall, R. Cipolla, Segnet: A deep convolutional encoder-decoder architecture for image segmentation, arXiv preprint
arXiv:1511.00561.
[54] S. Jégou, M. Drozdzal, D. Vazquez, A. Romero, Y. Bengio, The one hundred layers tiramisu: Fully convolutional densenets for semantic segmentation, arXiv preprint arXiv:1611.09326.

27

