Cystoid macular edema segmentation of Optical Coherence Tomography images using
fully convolutional neural networks and fully connected CRFs
Fangliang Bai, Manuel J. Marques, Stuart J. Gibson
School of Physical Sciences, University of Kent, Canterbury, CT2 7NH, UK.

arXiv:1709.05324v1 [] 15 Sep 2017

Abstract
In this paper we present a new method for cystoid macular edema (CME) segmentation in retinal Optical Coherence Tomography
(OCT) images, using a fully convolutional neural network (FCN) and a fully connected conditional random fields (dense CRFs).
As a first step, the framework trains the FCN model to extract features from retinal layers in OCT images, which exhibit CME,
and then segments CME regions using the trained model. Thereafter, dense CRFs are used to refine the segmentation according
to the edema appearance. We have trained and tested the framework with OCT images from 10 patients with diabetic macular
edema (DME). Our experimental results show that fluid and concrete macular edema areas were segmented with good adherence
to boundaries. A segmentation accuracy of 0.61 ± 0.21 (Dice coefficient) was achieved, with respect to the ground truth, which
compares favorably with the previous state-of-the-art that used a kernel regression based method (0.51 ± 0.34). Our approach is
versatile and we believe it can be easily adapted to detect other macular defects.
Keywords: Optical coherence tomography, diabetic macular edema, object segmentation, conditional random fields, fully
convolutional neural networks, transfer learning

1. Introduction
Optical coherence tomography (OCT) has been regarded as
one of the most important test instruments in ophthalmic diagnosis [1] over the last two decades. During the last decade,
the performance of OCT systems experienced a significant improvement, with several variants being introduced and developed further [2, 3]. Spectral-domain OCT (SD-OCT) is a pivotal technique which was first introduced in 2006, paving the
way for high-resolution and fast-acquisition OCT systems [4].
SD-OCT utilizes a broadband optical source and an interferometer, which is interrogated by a high-speed spectrometer. The
spectra detected exhibit modulations which are proportional
to the depth locations of the features inside the object under
study. This method can achieve high at scanning speeds, upwards of 300,000 A-scans per second [5]. In ophthalmic applications, SD-OCT is used as an ancillary test for imaging both
the anterior and the posterior part of the eye, covering regions
such as the cornea, macula, optical nerve, and the choroidal
layer [6]. Due to the capabilities of long-range imaging and
clear visualization enabled by SD-OCT, it has become straightforward to detect symptoms of age-related macular degeneration (AMD), diabetic macular edema (DME) and central serous
chorioretinopathy etc. [7, 8, 9, 10, 11].
Cystoid macular edema (CME) is the major cause of blindness in patients with diabetic retinopathy. This ocular manifes-

Email addresses: F.Bai@kent.ac.uk (Fangliang Bai),
M.J.Marques@kent.ac.uk (Manuel J. Marques),
S.J.Gibson@kent.ac.uk (Stuart J. Gibson)
Preprint submitted to Elsevier

tation is becoming a common health issue in ophthalmic disease [12]. According to research [13], approximately 25% of
patients with diabetes suffer from diabetic retinopathy and the
likelihood of developing CME increases the longer a patient
lives with diabetes. The morphology of diabetic retinopathy
has four classifications; diabetic macular edema (DME), retinal detachment, diffuse retinal thickening, and vitreomacular
interface abnormalities [14]. Apart from diabetic retinopathy,
CME is also a cause of visual degradation in other pathological
conditions, including AMD, vein occlusion, epiretinal membrane traction, etc. [15]. Since OCT is a non-invasive imaging method which allows the detection of CME in a reasonably
simple manner, many clinicians prefer the use of OCT for the
assessment of patients with diabetic retinopathy [16, 17]. The
advantage of using OCT is that it eases the quantitative assessment, thus presenting more structural information than the qualitative evaluation performed with fundus photography. Therefore, measuring the quantity of CME and monitoring its change
over time is of significant importance in clinical assessment.
Segmentation of CME in OCT images is always a challenging task for clinicians. OCT systems with high acquisition rates
yield large amounts of 3D data, consisting of hundreds of Bscan images in each volume. Manual annotation of each Bscan image would require significant processing time which is
impractical for clinical applications. Furthermore, when carrying out the quantitative assessment it is preferable to have the
segmentation in a 3D format, since this provides more morphological information. However, the imaging acquisition process presents deviations due to involuntary movements of eyes
and head of the subject. These movements affect continuity
September 18, 2017

and smoothness within interval frames of the OCT volumes.
This defect also introduces difficulties in manual annotation.
Apart from the difficulty mentioned above, a severe retinopathy can result in a high diffusion of boundaries between healthy
and diseased tissue. This diffusion is likely to cause inconsistencies of manual annotation results between different experts.
The manual segmentation of CME areas can therefore become
very subjective. Hence, various automated segmentation approaches have been developed to produce time-efficient and
stable detections yielding different parameters, such as corneal
angle, retinal layer boundaries, layer thickness and CME area
[18, 19, 20, 21, 22].
In this article, we propose an automated method to segment regions, exhibiting polymorphous cystoid macular edema,
in retinal OCT images using a deep-learning based approach
which has been used extensively in natural image tasks [23]. To
the best of our knowledge, our work is the first to employ fully
convolutional neural networks with dense conditional random
fields in segmentation tasks involving retinal OCT imagery.
The article is structured as following. In Section 2, the CME
manifestation , the existing methods, the fully convolutional
neural network and the fully connected CRFs are reviewed separately. In Section 3, we present the framework of CME segmentation on retinal OCT images. In Section 4, we present and
discuss the results. And in Section 5 we end the article with
conclusion and future work.

good agreement between the two manually segmented images
(B and C). At the end of this article, we will show that the proposed method can objectively outperform manual segmentation
in terms of consistency metrics.

Figure 2.1: A retinal OCT image with DME symptoms in the foveal region from
our experimental dataset. The retinal layers are located between the vitreous
and choroid layers. The highly reflective dots with shadows underneath are
blood vessels and their shadows. In the fovea, retinal layers are detached by a
mixture of fluid and concrete edema.

Figure 2.2: Manual segmentation of DME in OCT retinal images carried out by
two different graders: (A) the original image; (B) and (C) manual segmentation
performed by two different graders. There is a significant difference between
the two manually segmented images. The central region, whose appearance is
closer to a concrete edema, was annotated by the first grader (B) but not by the
second one (C). This variability may be caused by the OCT image quality, the
appearance of the edema and the graders experience.

2. Review
In this section, we briefly review the properties of CME
present in OCT images from subjects with diabetic retinopathy,
followed by a discussion of the existing methods for CME segmentation, the fully convolutional neural network (FCN) model
and the fully connected CRFs (Dense CRFs) procedure as a
post-process of refining segmentation.

2.2. Existing approaches for CME segmentation
With the development of OCT imaging systems, the task of
segmenting retinal structures has been deeply researched and
implemented with various approaches. However, most existing
segmentation methods focus on delineating retinal layer boundaries in healthy eyes. One branch of image segmentation methods are based on energy minimization techniques [24, 25]. This
group of algorithms represents the OCT image as an adjacency
graph and each pixel as the node of the graph. The edge of
the graph represents the degree of similarity between two connected nodes. Then graph cut theory [26] is used to minimize
the summation of the weights of graph edges with the aim of
finding the boundaries. A well developed method was proposed
by Chiu et al. [21] who used graph searching and dynamic programming to find the shortest path between left and right edges
of the graph. This method and its variants explore the characteristics of the retinal structure and calculate the graph path
based on the image gradient. This method can produce a decent segmentation result on normal retinal structures. However,
the segmentation accuracy decreases when performing the segmentation on pathological retinal structures, where the various
morphological conditions which deform the structures induce
complexity and lead to poorly delineated boundaries.

2.1. Character of Human CME in OCT image
An example of an OCT retinal image exhibiting DME is
shown in Fig. 2.1. The retinal layers are deformed and discrete
with large cystic fluid areas embedded within the sub-retinal
layers. The diabetic edema can be categorized into types: fluid
and concrete. The fluid regions, embedded in various shapes
and sizes, contain sparse tissue that exhibits low reflectivity.
Therefore the fluid regions in the OCT images are generally
shown as dark blocks surrounded by layers with high reflectivity. Conversely, the concrete regions are formed by denser
tissue. The concrete area is not easily distinguishable from
healthy tissue in these images, resulting in the boundary between healthy and diseased tissue not being clearly shown and
segmented. Conventionally, the medical professionals segment
the concrete area by subjectively estimating the thickness and
delineating the boundary at what might be a reasonable position based on their experience. Therefore, different professionals may produce inconsistent results. Figure 2.1 depicts annotations obtained from two graders (both medical professionals). For example, the delineation of edema does not have a
2

Some approaches toward edema fluid segmentation have
been implemented which are derivatives of methods which
segment layer boundaries. A fluid-filled region segmentation
method by Fernandez [27] adapted an active contour model to
segment individual fluid edema regions. The active contour
model is well designed to delineate the retinal lesion boundaries. It employs assumptions based upon a set of initial boundaries. It requires the region to be closed and considers the region to be homogeneous when compared to the surrounding
area. More recent work, developed by Chiu et al.[19], introduced kernel regression to a previously described graph searching framework to classify the layer boundaries and edema regions. In their proposed method, they adapted an iterative
Gaussian steering kernel to a supervised classification function
that uses local features to classify pixels into a range of categories. Graph theory and a dynamic programming algorithm
was used as a classification estimate to segment the retinal layer
and the edema regions. In the results they reported, this method
demonstrated good accuracy for layer segmentation but the accuracy for edema segmentation was relatively poor.
All the methods reviewed above, include some ad hoc rules
built into the mathematical models which are based on the nature of the target object. When the character of the target object
becomes varied and complex, more conditions should be considered to maintain the accuracy, which results in the model
becoming complicated, and less efficient. Additionally, with
more specifically designed rules, the application scope of the
approach is narrowed, making it not applicable to multi-purpose
diagnosis. In Section 4, we show that our approach can adapt to
various conditions and characteristics of the target object. We
compared our segmentation results with the results of Chiu et
al.[19] which indicates an improvement on edema region segmentation.

features such as appearance information from the image. However, these also involve dense computations when carrying on
into deep convolutional layers. To mitigate the high computational cost, the model inserts a pooling layer after each convolutional layer. The pooling layer reduces the dimension of feature
maps by fusing (maximum or average) local spatial information with the kernel [32]. After these operations are carried
out in the deep layers, the size of each feature map decreases.
The last layer of FCN is an up-sampling layer that operates a
transposed convolution (a.k.a deconvolution in Caffe [33]) on
the feature maps. The up-sampled feature maps are then fused
to generate a heat map that indicates the location of target object. The predicted results may be coarse due to the high downsampling rate through the network. This can be improved by
using the skip link which fuses the predictions from previous
pooling layers with the prediction from the final convolution
layer. For instance in Fig. 2.3, a 1 × 1 convolution layer is inserted after pool4 to produce additional class predictions. This
prediction is fused with the predictions from conv7 by adding
a 2× up-sampling layer and summing both predictions. This
up-sampling layer is initialized to be a bilinear interpolation of
conv7, however the parameters are learnable during training. It
was found that this strategy improves the prediction by fusing
back to the pool3.
During the training process, the weights for convolutional
layers can be those of the pre-trained model of modified classification networks rather than just randomly initialized. Then
the transfer learning of the neural network can be applied to
transferring learned features of the pre-trained network to learn
features in the new problem by fine-tuning the parameters during training. The training error between the predicted results
and the previously labeled images is calculated after each training batch. Then a back-propagation process [34] is utilized to
fine-tune the layer weights.

2.3. Fully convolutional neural network model
A fully convolutional neural network (FCN) [28] is an endto-end training model that uses a sequence of convolutional layers to extract and predict the class of pixels of the input image.
A common FCN model can be constructed by replacing a fully
connected layer of classification networks, such as VGG net
[29] and GoogleLeNet [30] , with a convolutional layer and using fine-tuning to exploit the learned convolutional kernel of the
classification networks for the segmentation task. The key point
of FCN is its skip architecture [31] that combines detailed information extracted from shallow convolutional layers with semantic information from deep convolutional layers. A common
FCN model is formed with a set of convolutional layers, pooling layers, transposed convolutional layers and skip links from
pooling layers to transposed convolutional layers. The general
architecture FCN [28] is schematically represented in Fig. 2.3.
Given a two dimensional input image, I, of size M × N × 1,
the first convolutional layer convolves the image with F1 spatial
kernels, each of which have a size of m1 × n1 × 1. The convolution for each kernel results in a feature map whose size depends
on kernel size and stride size. At the output of the first convolutional layer, the F1 feature maps are stacked as an input to
the next layer. Multiple kernel convolutions can extract diverse

2.4. Fully connected CRFs classification
For a given image and corresponding label prediction which
could be the heat map produced with the FCN model, the fully
connected CRFs model finely corrects the class for each pixel
based on feature vectors of image and heat map (the vector contains pixel location and value in different channels) and generates an amended prediction map as the final output. Considering an image I of size N and random field X over K possible pixel-level labels L = {l1 , l2 , · · · , lk }, a conditional random
field (I, X) is characterized by a Gibbs distribution P(X | I) =
P
1
Z(I)) exp(− c∈CG ψc (X | I)) where G = (V, E) is a graph on X in
which each clique c in a set of cliques CG induces a potential
ψc . In the fully connected CRFs model, G is a fully connected
graph of X and CG is the set of all unary and pairwise cliques
which forms the Gibbs energy function of labeling x ∈ LN in
Equation 1,
E(X) =

X

ψc (xc )

c∈CG

=

X
i

3

ψu (xi ) +

X
i< j

ψ p (xi , x j )

(1)

Figure 2.3: The general architecture of a fully convolutional neural network[28]. The convolutional layers with different filter size carry out the convolution
operations with the output of the previous layer or the initial image input. The convolutional layers are followed by the pooling layer that down-samples the
dimension of data while mitigating the computational overhead. The fusion link between the network output and that of previous convolutional layers fuses the
information of both deep and shallow layers, which refines the spatial segmentation of the image.

where i and j range from 1 to N. The unary potential ψu (xi ) is
calculated for each pixel by the FCN model, producing a heat
map with label assignment xi . The pairwise potential is described by:
ψ p (xi , x j ) = µ(xi , x j )

K
X

ω(m) k(m) ( fi , f j )

OCT volume. This algorithm is a block-matching de-noising
method using sparse 3D transform-domain collaborative filtering, with the aim of generating high contrast OCT images. The
de-noising process suppresses OCT speckle noise while sharpening the image, ultimately improving the presentation of CME
region. After the de-noising step, the retinal region is cropped
from the OCT volume to eliminate the unrelated vitreous objects and the tissues beneath the Choroidal layer.

(2)

m=1

where each k(m) is a set of Gaussian kernel functions for the feature vector f of pixel i and j separately, ω(m) describes the linear combination weights and µ is a label compatibility function
which introduces a penalty for neighboring similar pixels which
were assigned different labels. The pairwise function classifies pixels with similar feature vectors into the same class so
that different classes are prone to be divided at content boundaries. Then the fully connected CRFs model infers the final
label of each pixel by using mean field approximation, which is
described in detail in Koltun [35].

3.3. Network architecture of FCN model
The FCN-8 model adapted the architecture from the VGG16 network [29]. This architecture fuses the final convolutional
layer with the output from the last two pooling layers to make
an 8 pixel stride prediction net. We modified the network by
adding an additional score layer to reduce the prediction to 2
classes. We also used a dice layer to compute the overlap with
a dice similarity coefficient [37] between predicted results and
ground truth labels. By using the pre-trained weights of the convolutional kernel, we can transfer pre-trained weights to learn
new features of retinal OCT images.

3. Method

3.4. Data augmentation
The original data set only contained 110 OCT images. We
enriched the data by applying spatial translations to each image, which involved random translation, rotation, flipping and
cropping. The total number of training images was increased
to 2800 in this manner, and 20% of these images were used for
validation.

3.1. Problem statement
Given a retinal OCT image I M×N , the goal is to assign each
pixel {xi, j , i ∈ M, j ∈ N} to a class c where c is in the range of
the class space C = {c} = {1, · · · , K} for K classes. We consider
a 2-class problem that treats the CME regions as the target class
and others as background class.
3.2. Pre-processing
The Pre-processing of retinal OCT images involves 3D denoising and region cropping. Due to the speckle noise present
in the images, the BM3D algorithm [36] is used on each 3D

3.5. Loss function and optimization
The network is jointly trained with logistic loss and Dice loss.
The multinomial logistic loss, used in original FCN model [28],
4

provides a probabilistic similarity at the pixel level, comparing
the ground truth label and the prediction results. In our model,
the loss function was implemented using softmax with loss
layer, built in Caffe [33]. We also use Dice loss [37] to compute the spatial overlap between prediction and ground truth.
During the training process, we have used the stochastic gradient descent (SGD) as an optimizer to reduce the loss function.

4.3. Evaluation metrics
In order to compare our method with Chiu’s work, the evaluation of the framework is done based on same the two metrics
reported in [19]. With a manual segmentation by a grader as
ground truth, we measured the Dice overlap coefficient [40] for
the fluid segmentation. The Dice coefficient, defined in Equation 3, calculates the index of overlay of auto-segmented results
Xauto and manual annotation Xmanual . Furthermore, we calculated the mean Dice coefficient for each patient and performed
the Wilcoxon matched-pairs test across all patients to evaluate
the difference between framework results and the corresponding manual results.

4. Experiments
4.1. Experimental datasets
To validate the effectiveness of our approach, the method was
tested on the same dataset used in previous work by Chiu et al.
[19]. The dataset comprises 110 SD-OCT B-scan images which
are collected from 10 anonymous patients with DME. Selected
B-scans are located at the fovea and at two opposite sides of
the foveal region (1 B-scan at central fovea and 5 B-scans at 2,
5, 10, 15 and 20 slices away from the central region on each
side). These B-scan images are annotated for 7 retinal layers
and intra-layer fluid sections. The image size is 512 × 740. For
each patient, 11 B-scans of the 3D volume data are annotated by
two professional graders using DOCTRAP semi-automatic segmentation software (Duke University, Durham, NC, USA) [38]
or manually annotated. Specifically, the fluid CME section was
firstly annotated using DOCTRAP software package, then reviewed and manually corrected to refine the segmentation. For
the concrete regions, the grader performed the annotation from
experience by estimating the region in relation to the thickness
of the retinal layer.

Dice =

2 |Xauto ∩ Xmanual |
|Xauto | + |Xmanual |

(3)

4.4. Segmentation results
After training the FCN network, we tested the framework
with a separate test data set which was not previously seen by
the network. The selected data set comprises a set of OCT images exhibiting various morphologies of DME symptoms. In
this section we show the segmentation results of the framework
compared to manual segmentation results for different DME
conditions. We also present the comparison between manual
annotations and the framework results where small edema regions failed to be segmented. Throughout the whole study, our
results were compared with those from [19] in both a qualitative and quantitative way using the same metric to show an
improvement in terms of segmentation accuracy.
The appearance of fluid edema in OCT images is often shown
with a higher contrast against surrounding tissue. Figure 4.1 (A)
depicts the foveal region affected with a fluid edema. The dominant edema detaches retinal layers and small edema regions
surround it sporadically. It can be seen on the framework results that the boundaries of the dominant edema are accurately
segmented out, and the smaller edema on the right-hand side
is also segmented individually with clear boundaries. Although
two graders annotated the same fluid regions, the framework
shows an improvement in terms of boundary recall. A significant inconsistency among manual segmentations on the left side
of the fovea has also been observed. Because our network was
trained using labels previously provided by the first grader, it is
prone to be more similar to their results.
Figure 4.2 (A) shows a concrete edema embedded within
retinal layers on the right-hand side of the fovea region. The
distinctive features of this edema are not visually recognizable
in this OCT image due to low contrast within the retinal layers. However the presence of the edema causes a deformation
of the retinal layers on the right-hand side, with a corresponding change of the layer interval, making them noticeably thicker
than those on left-hand side. It is observed in Fig. 4.2 (D) that
the segmented results obtained with our approach presents a
high correspondence to the manually segmented results. However, it is worth noting that our approach has segmented several
regions which had been represented separately under manual
annotation as a single large one.

4.2. Experimental setting
In our experiments, we adapted the pre-trained FCN-8 model
from Hariharan et al. [39]. The architecture of the network is
shown in Table 1. The model weights were pre-tuned rather
than randomly initialized. A further fine-tuning was performed
to fit the OCT images. The hyperparameter settings for the
training are as follows: the number of training epochs is 30;
the batch size is set to 1; the base learning rate is 1.0e10−4 and
is reduced by an order of magnitude after every 10 epochs; and
the validation interval was done after each epoch. For the input batch, we keep the original pixel values rather than those
subtracted by the mean value. The experiments were run on a
desktop computer with an Intel CPU, 16 GB of RAM, and a
NVIDIA 1080Ti GPU with 16GB VRAM.
The grayscale OCT image data was reformatted to RGB
with channel conversion as required by the network architecture. The training data was stored using an lightning memorymapped database backend with PNG compression format for
feature encoding and label encoding. 20% of data was used for
validation process.
We also modified the parameter setting of the fully connected
CRFs model. The original model [35] was devised for RGB
image inference. In order to adapt the model for grayscale images, the parameters of the kernel were changed in the pairwise
function. The standard deviation is 2 for the color independent
function, and 0.01 for the color dependent term. We set the
ground truth certainty to 0.6 in order to obtain the best results.
5

Table 1: The architecture of the FCN-8 model used in the experiment.

Block 1
Block 2
Block 3
Block 4
Block 5
Block 6
Block 7
Block 8
Block 9
Block 10
Block 11
Block 12
Block 13
Block 14

Block 15

Block 16
Block 17
Block 17
Block 19

Type
Convolution
ReLU
Max Pool
Convolution
ReLU
Max Pool
Convolution
ReLU
Max Pool
Convolution
ReLU
Max Pool
Convolution
ReLU
Max Pool
Convolution
ReLU
Convolution
Deconvolution
Convolution
Crop
Element-wise Fuse
Deconvolution
Convolution
Crop
Element-wise Fuse
Deconvolution
Crop
Convolution
Dice
SoftmaxWithLoss

Filter size
3
2
3
2
3
2
3
2
3
2
7
1
4
1
4
1
16
1
-

Stride
1
2
1
2
1
2
1
2
1
2
1
1
2
1
2
1
8
1
-

Filter number
64
128
256
512
512
4096
21
21
21
21
21
21
2
-

In Fig. 4.3 (D) we demonstrate results of our framework operating on images depicting more severe morphological conditions. The retinal layer in Fig. 4.3 (A) was destroyed by
a large section of edema which mixed various shapes of fluid
and concrete edemas. The central fluid section is fused with
the concrete part on both the left and the right-hand side of the
images, where both sections were successfully segmented by
our approach. It can be seen that our approach shows that the
segmentation achieved is consistent with the appearance of the
pathology and accurately covers the entire region. Several small
sections were also well segmented but some slimmer region is
not detected since it is too slim to be perceived by the network.
In Fig. 4.5, we present additional results from severe conditions
that present complex structure appearances.

Padding
100
0
1
0
1
0
1
0
1
0
0
0
0
0
0
-

Number of blocks

Fusion Link

×2
×1
×2
×1
×3
×1
×3
×1
×3
×1
×2
×1

×1

×2 fusion

×1

×4 fusion

×1
×1
×1
×1

small initial size leading to them being smoothed out after several pooling layers. In another case, some target regions were
only partially segmented, as shown in Fig. 4.4.
Figure 4.5 depicts the comparison of our method with the one
reported in Chiu et al. [19] for three segmentation attempts carried out complex retinopathy conditions. It is found that Chiu’s
method performs a slight over-segmentation near the bottom
area. The segmentation results carried out with our method
are closer to those which have been manually segmented. In
Fig. 4.6, three images are shown which depict segmentation
results on concrete and non-obvious fluid edema. The oversegmentation noted earlier can also be observed , as well as
a false segmentation happening in the region on the left-hand
side. Again, the results obtained with our method have a better
agreement with those obtained via manual segmentation. Furthermore, our results are more adherent to target based on image content. This is also found in Fig. 4.7 where additional results are presented from the segmentation of an example of fluid
edema. Quantitatively, we compared the Dice overlap coefficient and p-value of Wilcoxon matched-pairs test for the DME
region. Dice coefficients were calculated for all 10 patients and

Apart from the segmentation of large areas, there are still
some cases where our approach failed to segment the target region accurately. One such event occurred when small target
regions were not segmented out, as shown in Fig. 4.1 (D). In
this case, the small regions were affected by speckle noise and
smoothing effects caused by the pre-processing resulting in the
small regions being undetectable by the framework, due to their
6

Figure 4.1: The segmentation results of fluid edema by graders and the framework. (A): original image of the fluid edema region. (B) and (C): segmented
results by two separate graders. The segmentation of the fluid region is in good
agreement, however the region on the left-hand side was not segmented by the
first grader. (D): segmented results obtained with the framework. The fluid sections are segmented from the boundary. On the left-hand side, a small region of
concrete edema is also detected. However unlike the image obtained from the
second grader, the entire region is not covered.

Figure 4.2: The segmentation results of concrete edema by graders and the
framework. (A): original image. (B) and (C): segmented results by two separate
graders. The segmentation results by two graders are similar. (D): segmented
results obtained with our framework. On the right-hand side, most of the region is segmented continuously. The segmented region just under the foveal pit
partially covers the small region of concrete edema at the fovea.

the process was repeated as per[19]. Specifically, the Dice coefficient was calculated based on all test images and the Wilcoxon
matched-pairs test was calculated based on the mean Dice coefficient across all patients for our automated method and for the
corresponding results from two graders. It was found that the
Dice coefficient of our approach is 0.61±0.21 standing for mean
and standard deviation which outperforms 0.51 ± 0.34 reported
in [19] (the higher, the better). Both methods are comparable
with the Dice coefficient between manual graders ( 0.58±0.32).
The p-value for our approach is 0.53, which is also better than
0.43 reported in Chiu et al. [19] (where a coefficient value of 1
indicates perfect agreement).

neural networks (FCN) followed by post-precessing involving
the fully connected CRFs (dense CRFs) method. The FCN
model is an end-to-end network that takes advantage of the convolutional layers with skipping links to predict tissue classification at the pixel level. The skipping link allows the output from
deeper convolutional layers to be fused with that from shallower
convolutional layers. This feature compensates for the coarse
prediction caused by down-sampling in the pooling layer and
produces more refined results. We added the Dice layer to train
the network where the overlap of predicted results and ground
truth was taken into account during training process. We further
fine-tuned the network using the pre-trained model and utilized
data augmentation to compensate for the requirement of large
amounts of training data. The dense CRFs are integrated to
further refine the segmentation results. With dense CRFs, the
recall rate of the boundaries was improved.

5. Conclusions and future work
In this paper, we presented a framework for segmenting CME
regions in OCT retinal images by training fully convolutional
7

Figure 4.4: The failed case of segmentation result. (A): original image. (B) and
(C): segmented results by two graders. (D): segmented results by the framework. It can be seen that part of the concrete edema is not completely segmented where it appears close to the surrounding tissue.

cal model-based approaches. Therefore, this study is valuable
for the final goal of developing a universal OCT segmentation
approach.
The results from our experiments also identified some disadvantages of the framework. Although the network architecture involves the skip link to fuse the appearance information
of shallow layers with the deeper layer outputs to make a more
precise prediction, the structural character still presents some
limitations in extracting accurate locations of small target regions. However, this disadvantage may be mitigated by modifying the network structure, namely by strengthening the endto-end training.
In this paper, our experiment is focused on segmentation of
cystoid macular edema. As future work, we intend to extend the
method to several ophthalmic conditions, including deformed
retinal layer segmentation caused by age-related macular degeneration, epiretinal membrane, macular hole and retinal detachment. We also intend to investigate the benefits of using
more efficient network architectures to improve the segmentation accuracy. At the same time, we plan to incorporate supervoxel and recurrent neural networks as a post-processing step
to enhance the inference efficiency with the aim of 3D segmentation which is more practical for quantitative analysis and clinical diagnosis.

Figure 4.3: Segmentation results for more severe edema as carried out by
graders and the framework. (A): original image. (B) and (C): segmented results
by two graders. (D): segmented results by the framework. It is seen that both
the concrete an fluid regions exhibiting complex appearances are segmented.

Acknowledgements
We appreciate the discussion and suggestion on framework architecture given by Jinchao Liu from Visionmetric Ltd,
and the technical support during the development for system
compatibility and hardware deployment from Nicholas French
(SPS, Kent). We also appreciate the suggestions and comments
about the manuscript from Sally Makin (SPS, Kent).

The segmentation results obtained with this framework
demonstrates the potential abilities of end-to-end convolutional
neural networks for OCT retinal image segmentation tasks. We
emphasize that a key advantage of this method it’s potential to
cope with a variety of symptoms. The morphological properties of a certain retinopathy can be diverse and differently represented depending on different conditions. Conventional segmentation methods for OCT retinal image segmentation were
often tailored for a specific task and their performance may vary
depending on retinopathy condition. Conversely, convolutional
neural networks can be trained to identify various retinopathy
conditions, thus avoiding the requirement for a multitude of ad
hoc rules. We intend to extend the application of this framework for retinal layer segmentation and segmentation of other
diseases by training the network with larger training datasets
via manual annotations of the symptoms of interest. It is expected that this learning based approach be adapted for more
complex cases and can be more flexible than fixed mathemati-

References
[1] D. Huang, E. A. Swanson, C. P. Lin, J. S. Schuman, W. G. Stinson,
W. Chang, M. R. Hee, T. Flotte, K. Gregory, C. A. Puliafito, et al., Optical
coherence tomography, Science (New York, NY) 254 (1991) 1178.
[2] A. Bradu, K. Kapinchev, F. Barnes, A. Podoleanu, Master slave en-face
oct/slo, Biomedical optics express 6 (2015) 3655–3669.
[3] M. J. Marquesa, S. Rivetb, A. Bradua, A. Podoleanua, Polarizationsensitive plug-in optical module for a fourier-domain optical coherence
tomography system, in: Proc. of SPIE Vol, volume 10053, pp. 100531B–
1.
[4] J. F. De Boer, B. Cense, B. H. Park, M. C. Pierce, G. J. Tearney, B. E.
Bouma, Improved signal-to-noise ratio in spectral-domain compared
with time-domain optical coherence tomography, Optics letters 28 (2003)
2067–2069.

8

Figure 4.5: A comparison of Chiu’s work and our results. (A) and (B): annotations by two graders. (C): Segmentation results from Chiu’s work. (D): our results.
From the comparison, it can be seen that the results at third row is slightly over-segmented and not coherent to the boundaries. This incoherence is not present in
our results.

Figure 4.6: A comparison between Chiu’s work and the results obtained with our method. (A) and (B): annotations by two graders. (C): Segmentation results from
Chiu’s work. (D): our results. From the comparison, it can be seen that both sub-segmentation and over-segmentation are corrected in our results which have a
higher agreement with manual annotation

[5] B. Potsaid, I. Gorczynska, V. J. Srinivasan, Y. Chen, J. Jiang, A. Cable,
J. G. Fujimoto, Ultrahigh speed spectral / fourier domain oct ophthalmic
imaging at 70,000 to 312,500 axial scans per second, Opt. Express 16
(2008) 15149–15169.
[6] M. Adhi, J. S. Duker, Optical coherence tomography–current and future

applications, Current opinion in ophthalmology 24 (2013) 213.
[7] V. Manjunath, M. Taha, J. G. Fujimoto, J. S. Duker, Choroidal thickness
in normal eyes measured using cirrus hd optical coherence tomography,
American journal of ophthalmology 150 (2010) 325–329.
[8] R. Margolis, R. F. Spaide, A pilot study of enhanced depth imaging op-

9

[24]
[25]

[26]

Figure 4.7: A comparison between Chiu’s work and the results obtained with
our method. (A) and (B): annotations by two graders. (C): Segmentation results
from Chiu’s work. (D): our results. It is clearly shown that our results are
adhering more consistently to the edema boundary of the manual segmentation.

[27]

[28]

[9]

[10]

[11]

[12]

[13]

[14]

[15]
[16]

[17]

[18]

[19]

[20]

[21]

[22]

[23]

tical coherence tomography of the choroid in normal eyes, American
journal of ophthalmology 147 (2009) 811–815.
A. E. Fung, G. A. Lalwani, P. J. Rosenfeld, S. R. Dubovy, S. Michels,
W. J. Feuer, C. A. Puliafito, J. L. Davis, H. W. Flynn, M. Esquiabro, An
optical coherence tomography-guided, variable dosing regimen with intravitreal ranibizumab (lucentis) for neovascular age-related macular degeneration, American journal of ophthalmology 143 (2007) 566–583.
B. Haouchine, P. Massin, R. Tadayoni, A. Erginay, A. Gaudric, Diagnosis
of macular pseudoholes and lamellar macular holes by optical coherence
tomography, American journal of ophthalmology 138 (2004) 732–739.
L. Yang, J. B. Jonas, W. Wei, Optical coherence tomography–assisted enhanced depth imaging of central serous chorioretinopathycentral serous
chorioretinopathy, Investigative ophthalmology & visual science 54
(2013) 4659–4665.
R. Klein, B. E. Klein, S. E. Moss, K. J. Cruickshanks, The wisconsin epidemiologic study of diabetic retinopathy: Xvii: The 14-year incidence and progression of diabetic retinopathy and associated risk factors in type 1 diabetes11proprietary interest: none., Ophthalmology 105
(1998) 1801–1815.
L. P. Aiello, R. L. Avery, P. G. Arrigg, B. A. Keyt, H. D. Jampel, S. T.
Shah, L. R. Pasquale, H. Thieme, M. A. Iwamoto, J. E. Park, et al., Vascular endothelial growth factor in ocular fluid of patients with diabetic
retinopathy and other retinal disorders, New England Journal of Medicine
331 (1994) 1480–1487.
T. Otani, S. Kishi, Y. Maruyama, Patterns of diabetic macular edema with
optical coherence tomography, American journal of ophthalmology 127
(1999) 688–693.
M. W. Johnson, Etiology and treatment of macular edema, American
journal of ophthalmology 147 (2009) 11–21.
I. Kozak, V. L. Morrison, T. M. Clark, D.-U. Bartsch, B. R. Lee, I. Falkenstein, A. M. Tammewar, F. Mojana, W. R. Freeman, Discrepancy between
fluorescein angiography and optical coherence tomography in detection
of macular disease, Retina (Philadelphia, Pa.) 28 (2008) 538.
M. R. Hee, C. A. Puliafito, C. Wong, J. S. Duker, E. Reichel, B. Rutledge,
J. S. Schuman, E. A. Swanson, J. G. Fujimoto, Quantitative assessment
of macular edema with optical coherence tomography, Archives of ophthalmology 113 (1995) 1019–1029.
P. A. Dufour, L. Ceklic, H. Abdillahi, S. Schroder, S. De Dzanet, U. WolfSchnurrbusch, J. Kowal, Graph-based multi-surface segmentation of oct
data using trained hard and soft constraints, IEEE transactions on medical
imaging 32 (2013) 531–543.
S. J. Chiu, M. J. Allingham, P. S. Mettu, S. W. Cousins, J. A. Izatt, S. Farsiu, Kernel regression based segmentation of optical coherence tomography images with diabetic macular edema, Biomedical optics express 6
(2015) 1172–1194.
P. P. Srinivasan, S. J. Heflin, J. A. Izatt, V. Y. Arshavsky, S. Farsiu, Automatic segmentation of up to ten layer boundaries in sd-oct images of the
mouse retina with and without missing layers due to pathology, Biomedical optics express 5 (2014) 348–365.
S. J. Chiu, X. T. Li, P. Nicholas, C. A. Toth, J. A. Izatt, S. Farsiu, Automatic segmentation of seven retinal layers in sdoct images congruent with
expert manual segmentation, Optics express 18 (2010) 19413–19428.
R. Kafieh, H. Rabbani, M. D. Abramoff, M. Sonka, Intra-retinal layer
segmentation of 3d optical coherence tomography using coarse grained
diffusion map, Medical image analysis 17 (2013) 907–928.
D. Martin, C. Fowlkes, D. Tal, J. Malik, A database of human segmented
natural images and its application to evaluating segmentation algorithms

[29]
[30]

[31]
[32]

[33]

[34]

[35]
[36]

[37]

[38]

[39]

[40]

10

and measuring ecological statistics, in: Proc. 8th Int’l Conf. Computer
Vision, volume 2, pp. 416–423.
Y. Boykov, G. Funka-Lea, Graph cuts and efficient nd image segmentation, International journal of computer vision 70 (2006) 109–131.
V. Kolmogorov, R. Zabin, What energy functions can be minimized via
graph cuts?, IEEE transactions on pattern analysis and machine intelligence 26 (2004) 147–159.
L. R. Ford, D. R. Fulkerson, Maximal flow through a network, Canadian
journal of Mathematics 8 (1956) 399–404.
D. C. Fernandez, Delineating fluid-filled region boundaries in optical
coherence tomography images of the retina, IEEE transactions on medical
imaging 24 (2005) 929–945.
J. Long, E. Shelhamer, T. Darrell, Fully convolutional networks for semantic segmentation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431–3440.
K. Simonyan, A. Zisserman, Very deep convolutional networks for largescale image recognition, arXiv preprint arXiv:1409.1556 (2014).
C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan,
V. Vanhoucke, A. Rabinovich, Going deeper with convolutions, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1–9.
C. M. Bishop, Pattern recognition and machine learning, springer, 2006.
P. Liskowski, K. Krawiec, Segmenting retinal blood vessels with¡? pub
newline?¿ deep neural networks, IEEE transactions on medical imaging
35 (2016) 2369–2380.
Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick,
S. Guadarrama, T. Darrell, Caffe: Convolutional architecture for fast feature embedding, arXiv preprint arXiv:1408.5093 (2014).
Y. LeCun, B. E. Boser, J. S. Denker, D. Henderson, R. E. Howard,
W. E. Hubbard, L. D. Jackel, Handwritten digit recognition with a backpropagation network, in: Advances in neural information processing systems, pp. 396–404.
V. Koltun, Efficient inference in fully connected crfs with gaussian edge
potentials, Adv. Neural Inf. Process. Syst 4 (2011).
K. Dabov, A. Foi, V. Katkovnik, K. Egiazarian, Bm3d image denoising
with shape-adaptive principal component analysis, in: SPARS’09-Signal
Processing with Adaptive Sparse Structured Representations.
F. Milletari, N. Navab, S.-A. Ahmadi, V-net: Fully convolutional neural networks for volumetric medical image segmentation, in: 3D Vision
(3DV), 2016 Fourth International Conference on, IEEE, pp. 565–571.
J. Y. Lee, S. J. Chiu, P. P. Srinivasan, J. A. Izatt, C. A. Toth, S. Farsiu, G. J.
Jaffe, Fully automatic software for retinal thickness in eyes with diabetic
macular edema from images acquired by cirrus and spectralis systemsfully automatic sd-oct retinal thickness software in dme, Investigative
ophthalmology & visual science 54 (2013) 7595–7602.
B. Hariharan, P. Arbelaez, L. Bourdev, S. Maji, J. Malik, Semantic contours from inverse detectors, in: International Conference on Computer
Vision (ICCV).
T. Sørensen, A method of establishing groups of equal amplitude in plant
sociology based on similarity of species and its application to analyses of
the vegetation on danish commons, Biol. Skr. 5 (1948) 1–34.

