abstract: Colorectal adenocarcinoma originating in intestinal glandular structures is the most common form of colon cancer. In clinical practice, the morphology of intestinal glands, including architectural appearance and glandular formation, is used by pathologists to inform prognosis and plan the treatment of individual patients. However, achieving good inter-observer as well as intra-observer reproducibility of cancer grading is still a major challenge in modern pathology. An automated approach which quantifies the morphology of glands is a solution to the problem.
  .
  This paper provides an overview to the Gland Segmentation in Colon Histology Images Challenge Contest (GlaS) held at MICCAI'2015. Details of the challenge, including organization, dataset and evaluation criteria, are presented, along with the method descriptions and evaluation results from the top performing methods.
author: Korsuk Sirinukunwattana[^1]; Josien P. W. Pluim; Hao Chen; Xiaojuan Qi; Pheng-Ann Heng; Yun Bo Guo; Li Yang Wang; Bogdan J. Matuszewski; Elia Bruni; Urko Sanchez; Anton Böhm; Olaf Ronneberger; Bassem Ben Cheikh; Daniel Racoceanu; Philipp Kainz; Michael Pfeiffer; Martin Urschler; David R. J. Snead; Nasir M. Rajpoot[^2][^3]
bibliography: main_preprint_arxiv_2016_08_30.bib
title: Gland Segmentation in Colon Histology Images: The GlaS Challenge Contest

***Index terms—*** Histology Image Analysis, Segmentation, Colon Cancer, Intestinal Gland, Digital Pathology

# Introduction

Cancer grading is the process of determining the extent of malignancy and is one of the primary criteria used in clinical practice to inform prognosis and plan the treatment of individual patients. However, achieving good reproducibility in grading most cancers remains one of the challenges in pathology practice . With digitized images of histology slides becoming increasingly ubiquitous, digital pathology offers a viable solution to this problem . Analysis of histology images enables extraction of quantitative morphological features, which can be used for computer-assisted grading of cancer making the grading process more objective and reproducible than it currently is . This has led to the recent surge in development of algorithms for histology image analysis.

In colorectal cancer, morphology of intestinal glands including architectural appearance and gland formation is a key criterion for cancer grading . Glands are important histological structures that are present in most organ systems as the main mechanism for secreting proteins and carbohydrates. An intestinal gland (colonic crypt) found in the epithelial layer of the colon, is made up of a single sheet of columnar epithelium, forming a finger-like tubular structure that extends from the inner surface of the colon into the underlying connective tissue . There are millions of glands in the human colon. Intestinal glands are responsible for absorption of water and nutrients, secretion of mucus to protect the epithelium from a hostile chemical and mechanical environment , as well as being a niche for epithelial cells to regenerate . Due to the hostile environment, the epithelial layer is continuously regenerating and is one of the fastest regenerating surface in human body . This renewal process requires coordination between cell proliferation, differentiation, and apoptosis. The loss of integrity in the epithelial cell regeneration, through a mechanism that is not yet clearly understood, results in colorectal adenocarcinoma, the most common type of colon cancer.

Manual segmentation of glands is a laborious process. Automated gland segmentation will allow extraction of quantitative features associated with gland morphology from digitized images of CRC tissue slides. Good quality gland segmentation will pave the way for computer-assisted grading of CRC and increase the reproducibility of cancer grading. However, consistent good quality gland segmentation for all the differentiation grades of cancer has remained a challenge. This was a main reason for organizing this challenge contest.

The Gland Segmentation in Colon Histology Images (GlaS) challenge[^4] brought together computer vision and medical image computing researchers to solve the problem of gland segmentation in digitized images of Hematoxylin and Eosin (H&E) stained tissue slides. Participants developed gland segmentation algorithms, which were applied to benign tissue and to colonic carcinomas. A training dataset was provided, together with ground truth annotations by an expert pathologist. The participants developed and optimized their algorithms on this dataset. The results were judged on the performance of the algorithms on test datasets. Success was measured by how closely the automated segmentation matched the pathologist's.

# Related Work

Recent papers indicate the increasing interest in histology image analysis applied to intestinal gland segmentation. In this section, we review some of these methods.

presented a region growing method, which first thresholds an image, in order to separate nuclei from other tissue components. Large empty regions, which potentially correspond to lumen found in the middle of glands, are then used to initialize the seed points for region growing. The expanding process for each seed is terminated when a surrounding chain of epithelial nuclei is reached, and subsequently false regions are removed. Although this method performs well in segmenting healthy and benign glands, it is less applicable to cancer cases, where the morphology of glands can be substantially deformed.

In contrast to the above method, which mainly uses pixel-level information, represented each tissue component as a disk. Each disk is represented by a vertex of a graph, with nearby disks joined by an edge between the corresponding vertices. They proposed an algorithm, using graph connectivity to identify initial seeds for region growing. To avoid an excessive expansion beyond the glandular region, caused, for example, by large gaps in the surrounding epithelial boundary, edges between nuclear objects are used as a barrier to halt region growing. Those regions that do not show glandular characteristics are eliminated at the last step. The validation of this method was limited only to the dataset with healthy and benign cases.

introduced a segmentation algorithm based on polar coordinates. A neighborhood of each gland and a center chosen inside the gland were considered. Using this center to define polar coordinates, the neighborhood is displayed in $(r,\theta)$ coordinates with the $r$-axis horizontal and the $\theta$-axis vertical. One obtains a vertical strip, periodic with period $2\pi$ in the vertical direction. As a result, the closed glandular boundary is transformed into an approximately vertical periodic path, allowing fast inference of the boundary through a conditional random field model. Support vector regression is later deployed to verify whether the estimated boundary corresponds to the true boundary. The algorithm performs well in both benign and malignant cases stained by Hematoxylin and DAB. However, the validation on routine H&E stained images was limited only to healthy cases.

recently formulated a segmentation approach based on Bayesian inference, which allows prior knowledge of the spatial connectivity and the arrangement of neighboring nuclei on the epithelial boundary to be taken into account. This approach treats each glandular structure as a polygon made of a random number of vertices. The idea is based on the observation that a glandular boundary is formed from closely arranged epithelial nuclei. Connecting edges between these epithelial nuclei gives a polygon that encapsulates the glandular structure. Inference of the polygon is made via Reversible-Jump Markov Chain Monte Carlo. The approach shows favorable segmentation results across all histologic grades (except for the undifferentiated grade) of colorectal cancers in H&E stained images. This method is slow but effective.

Most of the works for intestinal gland segmentation have used different datasets and/or criteria to assess their algorithms, making it difficult to objectively compare their performance. There have been many previous initiatives that provided common datasets and evaluation measures to validate algorithms on various medical imaging modalities . This not only allows a meaningful comparison of different algorithms but also allows the algorithms to be implemented and configured thoroughly to obtain optimal performance . Following these successful initiatives, we therefore organized the Gland Segmentation in Colon Histology Images (GlaS) challenge. This challenge was a first attempt to address the issues of reproducibility and comparability of the results of intestinal gland segmentation algorithms. It was also aimed at speeding up even further the development of algorithms for gland segmentation. Note that none of above methods for intestinal gland segmentation participated in this competition.

# Materials

```latex
\resizebox{0.8\columnwidth}{!}{
\begin{tabular}{|c|c|c|c|}
\hline
\multirow{2}{*}{Histologic Grade} & \multicolumn{3}{c|}{Number of Images (Width x Height in Pixels)} \\ \cline{2-4}
                                  & Training Part   & Test Part A   & Test Part B   \\ \hline
Benign                            &    $37\; \begin{cases} \;1&(574\times433)\\\;1&(589\times453)\\ \;35&(775\times522)\end{cases}$             &  $33\;\begin{cases} \;1&(574\times433) \\ \;4&(589\times453)\\ \;28&(775\times522) \end{cases}$             & $4\;(775\times522)$   \\ \hline
Malignant                        &  $48\;\begin{cases} \;1&(567\times430)\\\;3&(589\times453)\\\;44&(775\times522)\end{cases}$                &   $27\; \begin{cases} \;1&(578\times433)\\\;2&(581\times442)\\\;24&(775\times522) \end{cases}$            & $16\;(775\times522)$  \\ \hline
\end{tabular}}
```

The dataset used in this challenge consists of 165 images derived from 16 H&E stained histological sections of stage T3 or T4[^5] colorectal adenocarcinoma. Each section belongs to a different patient, and sections were processed in the laboratory on different occasions. Thus, the dataset exhibits high inter-subject variability in both stain distribution and tissue architecture. The digitization of these histological sections into whole-slide images (WSIs) was accomplished using a Zeiss MIRAX MIDI Slide Scanner with a pixel resolution of $0.465 \mu m$. The WSIs were subsequently rescaled to a pixel resolution of $0.620 \mu m$ (equivalent to $20\times$ objective magnification).

A total of 52 visual fields from both malignant and benign areas across the entire set of the WSIs were selected in order to cover as wide a variety of tissue architectures as possible. An expert pathologist (DRJS) then graded each visual field as either 'benign' or 'malignant', according to the overall glandular architecture. The pathologist also delineated the boundary of each individual glandular object on that visual field. We used this manual annotation as ground truth for automatic segmentation. Note that different glandular objects in an image may be part of the same gland. This is because a gland is a 3-dimensional structure that can appear as separated objects on a single tissue section. The visual fields were further separated into smaller, non-overlapping images, whose histologic grades (i.e. benign or malignant) were assigned the same value as the larger visual field. Representative example images of the two grades can be seen in Figure . This dataset was also previously used in the gland segmentation study by .

In the challenge, the dataset was separated into **Training Part**, **Test Part A**, and **Test Part B**. Note that the data were stratified according to the histologic grade and the visual field before splitting. This was done to ensure that none of the images from the same visual field appears in different parts of the dataset (i.e. Training, Test Part A, or Test Part B). However, since the data were not stratified based on patient, different visual fields from the same slide can appear in different parts of the dataset. A breakdown of the details of the dataset is shown in Table . The ground truth as well as the histologic grade which reflects morphology of glandular structures were provided for every image in the Training Part at the time of release. We used Test Part A and Test Part B as off-site and on-site test datasets respectively. Furthermore, to ensure blindness of evaluation, the ground truth and histologic grade of each image in the test parts were not released to the participants.

# Challenge Organization

The GlaS challenge contest was officially launched by the co-organizers (KS, JPWP, DRJS, NMR) on April 21$^\text{st}$, 2015, and was widely publicized through several channels. At the same point, a challenge website[^6] was set up to disseminate challenge-related information and to serve as a site for registration, submission of results, and communication between the organizers and contestants. The challenge involved 4 stages, as detailed below:

#### Stage 1: Registration and Release of the Training Data

The registration was open for a period of about two months (April 21$^\text{st}$ to June 30$^\text{th}$, 2015). Interested individuals or groups of up to 3 people that were affiliated with an academic institute or an industrial organization could register and download the training data (Training Part, see Section for details) to start developing their gland segmentation algorithms. From this point forward, we will refer to a separate individual or a group of registrants as a 'team'.

#### Stage 2: Submission of a Short Paper

In order to gain access to the first part of the test data, each registered team was required to submit a 2-page document containing a general description of their segmentation algorithms and some preliminary results obtained from running each algorithm on the training data. Each team could submit up to 3 different methods. The intention of this requirement was for the organizers to identify teams who were serious about participating in the challenge. The organizers based their reviews on two criteria: clarity of the method description and soundness of the validation strategy. Segmentation performance was not considered in this review. The submission of this document was due by July 17$^\text{th}$, 2015.

#### Stage 3: Release of the Test Data Part A and Submission of Segmentation Results

The first part of the test data (Test Part A, see Section for details) was released on August 14$^\text{th}$, 2015 to those teams selected from the previous stage which also agreed to participate in the GlaS contest. The teams were given a month to further adjust and optimize their segmentation algorithms, and carry out segmentation on Part A of the test data. Each team could hand-in up to 3 sets of results per method submitted in Stage 2. The submission of the segmentation results was due by September 14$^\text{th}$, 2015. Evaluation of the submitted results was not disclosed to the teams until after the challenge event.

#### Stage 4: GlaS'2015 Challenge Event

The event was held in conjunction with MICCAI'2015 on October 5$^\text{th}$, 2015. All teams were asked to produce segmentation results on the second part of the test data (Test Part B, see Section ) within 45 minutes. The teams could either bring their own machines or conduct an experiment remotely. There was no restriction on the number of machines that the teams could use to produce results. Those teams that could not be present at the event provided implementations of their algorithms with which the organizers carried out the segmentation on their behalf. Each team was also asked to give a short presentation, discussing their work. At the end of the event, the complete evaluation of segmentation results across both parts of the test data was announced, which included a final ranking of the submitted methods. This information is also available on the challenge website.

## Challenge Statistics

By the end of Stage 1, a total of 110 teams from different academic and industrial institutes had registered. A total of 21 teams submitted the 2-page document for review in Stage 2, and 20 teams were invited to participate in the GlaS competition event. In Stage 3, only 13 teams submitted results on Part A of the test data in time. Late entries were neither evaluated nor considered in the next stage of the competition. On the day of the challenge event, 11 of the 13 teams that submitted the results on time in Stage 3 attended the on-site competition and presented their work. The organizers carried out the segmentation on behalf of the other two teams that could not be present.

# Evaluation

The performance of each segmentation algorithm was evaluated based on three criteria: 1) accuracy of the detection of individual glands; 2) volume-based accuracy of the segmentation of individual glands; and 3) boundary-based similarity between glands and their corresponding segmentation. It may seem that volume-based segmentation accuracy would entail boundary-based segmentation accuracy between a gland and its segmentation. However, in practice, this is not always the case. The volume-based metric for segmentation accuracy used in this challenge, was defined and calculated using the label that the algorithm had assigned to each pixel, but the boundary-based metric used the position assigned by the algorithm to the boundary of each gland. Pixels labels may be fairly accurate, while the boundary curves are very different. The remainder of this section describes all metrics employed in the evaluation.

We use the concept of a pair of corresponding segmented and ground truth objects as proposed in . Let $\mathcal{S}$ denote a set of all segmented objects and $\mathcal{G}$ denote a set of all ground truth objects. We also include in each of these sets the empty object $\emptyset$. We define a function $G_*:\mathcal{S}\to \mathcal{G}$, by setting, for each segmented object $S\in \mathcal{S}$, $G_*(S) = G \in \mathcal{G}$ where $G$ has the largest possible overlapping area with $S$. Although there could be more than one $G \in \mathcal{G}$ that maximally overlaps $S$, this in practice is extremely rare, and it is good enough to consider one of these $G$ as the value of $G_*(S)$. If there is no overlapping $G$, we set $G_*(S) = \emptyset$. (However, in the context of Hausdorff distance – see Section – $G_*$ will be extended in a different way.) Similarly, we define $S_*:\mathcal{G}\to\mathcal{S}$, by setting, for each $G\in\mathcal{G}$, $S_*(G)=S\in\mathcal{S}$, where $S$ has the largest possible overlapping area with $G$. Note that $G_*$ and $S_*$ are, in general, neither injective, nor surjective. Nor are they inverse to each other, in general. They do, however, assign to each $G$ an $S=S_*(G)$, and to each $S$ a $G=G_*(S)$.

## Detection Accuracy

The F1 score is employed to measure the detection accuracy of individual glandular objects. A segmented glandular object that intersects with at least 50% of its ground truth object is counted as true positive, otherwise it is counted as false positive. The number of false negatives is calculated as the difference between the number of ground truth objects and the number of true positives. Given these definitions, the F1 score is defined by $$\label{eq:f1score}
\mathrm{F1score} = \frac{2\cdot \mathrm{Precision} \cdot \mathrm{Recall}}{\mathrm{Precision} + \mathrm{Recall}},$$ where $$\mathrm{Precision} = \frac{\mathrm{TP}}{\mathrm{TP} + \mathrm{FP}}, \quad \mathrm{Recall} = \frac{\mathrm{TP}}{\mathrm{TP} + \mathrm{FN}},$$ and $\mathrm{TP}, \mathrm{FP}$, and $\mathrm{FN}$ denote respectively the number of true positives, false positives, and false negatives from all images in the dataset.

## Volume-Based Segmentation Accuracy

### Object-Level Dice Index

The Dice index is a measure of agreement or similarity between two sets of samples. Given $G$, a set of pixels belonging to a ground truth object, and $S$, a set of pixels belonging to a segmented object, the Dice index is defined as follows: $$\label{eq:dice}
\mathrm{Dice}(G,S) = \frac{2 |G \cap S|}{|G| + |S|},$$ where $|\cdot|$ denotes set cardinality. The index ranges over the interval $[0,1]$, where the higher the value, the more concordant the segmentation result and the ground truth. A Dice index of 1 implies a perfect agreement. It is conventional that the segmentation accuracy on an image is calculated by $\mathrm{Dice}(G_\text{all},S_\text{all})$, where $G_\text{all}$ denotes the set of pixels of all ground truth objects and $S_\text{all}$ denotes the set of pixels of all segmented objects. The calculation made in this way measures the segmentation accuracy only at the pixel level, not at the gland level, which was the main focus of the competition.

To take the notion of an individual gland into account, we employ the object-level Dice index . Let $n_\mathcal{G}$ be the number of non-empty ground truth glands, as annotated by the expert pathologist. Similarly let $n_\mathcal{S}$ be the number of glands segmented by the algorithm, that is the number of non-empty segmented objects. Let $G_i \in \mathcal{G}$ denote the $i^\text{th}$ ground truth object, and let $S_j \in \mathcal{S}$ denote the $j^\text{th}$ segmented object. The object-level Dice index is defined as $$\label{eq:obj-dice}
\mathrm{Dice}_\mathrm{obj}(\mathcal{G},\mathcal{S}) = \frac{1}{2}\left[\sum_{i= 1}^{n_\mathcal{G}}\gamma_i \mathrm{Dice}(G_i,S_*(G_i)) + \sum_{j=1}^{n_\mathcal{S}} {\sigma}_j \mathrm{Dice}(G_*(S_j),S_j)\right],$$ where $$\gamma_i = |G_i| / \sum_{p=1}^{n_\mathcal{G}} |G_p|,  \quad {{\sigma}}_j = |S_j|/\sum_{q = 1}^{n_\mathcal{S}}|S_q|$$ On the right hand side of , the first summation term reflects how well each ground truth object overlaps its segmented object, and the second summation term reflects how well each segmented object overlaps its ground truth objects. Each term is weighted by the relative area of the object, giving less emphasis to small segmented and small ground truth objects.

In the competition, the object-level Dice index of the whole test dataset was calculated by including all the ground truth objects from all images in $\mathcal{G}$ and all the segmented objects from all images in $\mathcal{S}$.

### Adjusted Rand Index

We also included the adjusted Rand index as another evaluation measure of segmentation accuracy. This index was used for additional assessment of the algorithm performance in Section .

The adjusted Rand index measures similarity between the set of all ground truth objects $\mathcal{G}$ and the set of all segmented objects $\mathcal{S}$, based on how pixels in a pair are labeled. Two possible scenarios for the pair to be concordant are that (i) they are placed in the same ground truth object in $\mathcal{G}$ and the same segmented object in $\mathcal{S}$, and (ii) they are placed in different ground truth objects in $\mathcal{G}$ and in different segmented objects in $\mathcal{S}$. Define $n_{ij}$ as the number of pixels that are common to both the $i^\text{th}$ ground truth object and the $j^\text{th}$ segmented object, $n_{i,\cdot}$ as the total number of pixels in the $i^\text{th}$ ground truth object, $n_{\cdot,j}$ as the total number of pixels in the $j^\text{th}$ segmented object, and $n$ as the total number of pixels. Following a simple manipulation, it can be shown that the probability of agreement is equal to $$P_\text{agreement} = \left[\binom{n}{2} + 2\sum_{i=1}^{n_\mathcal{G}}\sum_{j=1}^{n_\mathcal{S}} \binom{n_{ij}}{2} - \sum_{i = 1}^{n_\mathcal{G}}\binom{n_{i,\cdot}}{2} - \sum_{j=1}^{n_\mathcal{S}}\binom{n_{\cdot,j}}{2}  \right]\Bigg/\binom{n}{2}.$$ Here, the numerator term corresponds to the total number of agreements, while the denominator term corresponds to the total number of all possible pairs of pixels. Under the assumption that the partition of pixels into ground truth objects in $\mathcal{G}$ and segmented objects in $\mathcal{S}$ follows a generalized hypergeometric distribution, the adjusted Rand index can be formulated as $$\mathrm{ARI}(\mathcal{G},\mathcal{S}) = \frac{\sum_{i=1}^{n_\mathcal{G}} \sum_{j=1}^{n_\mathcal{S}}\binom{n_{i,j}}{2} - \sum_{i=1}^{n_\mathcal{G}}\binom{n_i}{2}\sum_{j=1}^{n_\mathcal{S}}\binom{n_{\cdot,j}}{2}\big/\binom{n}{2}}{\frac{1}{2}\left[\sum_{i=1}^{n_\mathcal{G}} \binom{n_{i,\cdot}}{2} + \sum_{j=1}^{n_\mathcal{S}} \binom{n_{\cdot,j}}{2} \right] - \sum_{i=1}^{n_\mathcal{G}} \binom{n_{i,\cdot}}{2}\sum_{j=1}^{n_\mathcal{S}}\binom{n_{\cdot,j}}{2} \big/ \binom{n}{2} }.$$ The adjusted Rand index is bounded above by 1, and it can be negative.

## Boundary-Based Segmentation Accuracy

We measure the boundary-based segmentation accuracy between the segmented objects in $\mathcal{S}$ and the ground truth objects in $\mathcal{G}$ using the object-level Hausdorff distance. The usual definition of a Hausdorff distance between ground truth object $G$ and segmented object $S$ is $$\label{eq:hausdorff}
\mathrm{H}(G,S) = \max\{\sup_{x\in G} \inf_{y \in S} d(x, y), \sup_{y\in S} \inf_{x\in G} d(x,y)\}$$ where $d(x,y)$ denotes the distance between pixels $x\in G$ and $y\in S$. In this work, we use the Euclidean distance. According to , Hausdorff distance is the most extreme value from all distances between the pairs of nearest pixels on the boundaries of $S$ and $G$. Thus, the smaller the value of the Hausdorff distance, the higher the similarity between the boundaries of $S$ and $G$, and $S=G$ if their Hausdorff distance is zero.

To calculate the overall segmentation accuracy between a pair of corresponding segmented and ground truth objects, we now introduce object-level Hausdorff distance by imitating the definition of object-level Dice index . The object-level Hausdorff distance is defined as $$\mathrm{H}_\text{obj}(\mathcal{G},\mathcal{S}) = \frac{1}{2}\left[\sum_{i=1}^{n_\mathcal{G}} \gamma_i \mathrm{H}(G_i,S_*(G_i)) + \sum_{j=1}^{n_\mathcal{S}}{\sigma}_j \mathrm{H}(G_*(S_j),S_j)\right],$$ where the meaning of the mathematical notation is similar to that given in Section . In case a ground truth object $G$ does not have a corresponding segmented object (i.e. $S_*(G) = \emptyset$), the Hausdorff distance is calculated between $G$ and the nearest segmented object $S\in\mathcal{S}$ to $G$ (in the Hausdorff distance) in that image instead. The same applies for a segmented object that does not have a corresponding ground truth object.

# Ranking Scheme

Each submitted entry was assigned one ranking score per evaluation metric and set of test data. Since there were 3 evaluation metrics (F1 score for gland detection, object-level Dice index for volume-based segmentation accuracy, and object-level Hausdorff index for boundary-based segmentation accuracy) and 2 sets of test data, the total number of ranking scores was 6. The best performing entry was assigned ranking score 1, the second best was assigned ranking score 2, and so on. In care of a tie, the standard competition ranking was applied. For instance, F1 score 0.8, 0.7, 0.7, and 0.6 would result in the ranking scores 1, 2, 2, and 4. The final ranking was then obtained by adding all 6 ranking scores (rank sum). The entry with smallest sum was placed top in the final ranking.

# Methods

The top ranking methods are described in this section. They are selected from the total of 13 methods that participated in all stages of the challenge. The cut-off for the inclusion in this section was made where there was a substantial gap in the rank sums (see , Figure ). Of the 7 selected methods, only 6 preferred to have their methods described here.

## CUMedVision[^7]

A novel deep contour-aware network was presented. This method explored the multi-level feature representations with fully convolutional networks (FCN) . The network outputted segmentation probability maps and depicted the contours of gland objects simultaneously. The network architecture consisted of two parts: a down-sampling path and an up-sampling path. The down-sampling path contained convolutional and max-pooling layers while the up-sampling path contained convolutional and up-sampling layers, which increased the resolutions of feature maps and outputted the prediction masks. In total, there were 5 max-pooling layers and 3 up-sampling layers. Each layer with learned parameters was followed by a non-linear mapping layer (element-wise rectified linear activation).

In order to separate touching glands, the feature maps from hierarchical layers were up-sampled with two different branches to output the segmented object and contour masks respectively. The parameters of the down-sampling path were shared and updated for these two kinds of masks. This could be viewed as a multi-task learning framework with feature representations, simultaneously encoding the information of segmented objects and contours. To alleviate the problem of insufficient training data , an off-the-shelf model from DeepLab , trained on the 2012 PASCAL VOC dataset[^8], was used to initialize the weights for layers in the down-sampling path. The parameters of the network were obtained by minimizing the loss function with standard back-propagation [^9].

The team submitted two entries for evaluation. **CUMedVision1** was produced by FCN with multi-level feature representations relying only on gland object masks, while **CUMedVision2** was the results of the deep contour-aware network, which considers gland object and contour masks simultaneously.

## CVML[^10]

In the first, preprocessing, stage the images were corrected to compensate for variations in the appearance due to a variability of the tissue staining process. This was implemented through histogram matching, where the target histogram was calculated from the whole training data, and the individual image histograms were used as inputs. The main processing stage was based on two methods: a convolutional neural network (CNN) for a supervised pixel classification, and a level set segmentation for grouping pixels into spatially coherent structures. The employed CNN used an architecture with two convolutional, pooling and fully connected layers. The network was trained with three target classes. The classes were designed to represent (1) the tubular interior of the glandular structure (inner class), (2) epithelial cells forming boundary of the glandular structure (boundary class) and (3) inter-gland tissue (outer class). The inputs to the CNN were $19\times19$ pixel patches sliding across the adjusted RGB input image. The two convolutional layers used $6\times6$ and $4\times4$ kernels with 16 and 36 feature maps respectively. The pooling layers, implementing the mean function, used $2\times2$ receptive fields and $2\times2$ stride. The first and second fully connected layers used the rectified linear unit and softmax functions respectively. The outputs from the CNN were two probability maps representing the probability of each image pixel belonging to the inner and boundary classes. These two probability maps were normalized between -1 and 1 and used as a propagation term, along with an advection term and a curvature flow term. These terms were part of the hybrid level set model described in . In the post-processing stage, a sequence of morphological operations was performed to removed small objects, fill holes and disconnect weakly connected objects. Additionally, if an image boundary intersecting an object forms a hole, the corresponding pixels was labeled as part of that object. The team submitted a single entry for evaluation, henceforth referred to as **CVML**.

## ExB[^11]

This method first preprocessed the data by performing per channel zero mean and unit variance normalization, where the mean and variance were computed from the training data. The method then exploited the local invariance properties of the task by applying a set of transformations to the data. At training time, the dataset was augmented by applying affine transformations, Gaussian blur and warping. During testing, both image mirroring and rotation were applied.

The main segmentation algorithm consisted of a multi-path convolutional neural network. Each path was equipped with a different set of convolutional layers and configured to capture features from different views in a local-global fashion. All the different paths were connected to a set of two fully connected layers. A leaky rectified linear unit was used as a default activation function between layers, and a softmax layer was used after the last fully connected layer. Every network was trained via stochastic gradient descent with momentum, using a step-wise learning rate schedule . The network was randomly initialized such that unit variance was preserved across layers. It was found that using more than three paths led to heavy over-fitting – this was due to insufficient training data.

Simple-path networks were trained to detect borders of glands. The ground truth for these networks was constructed using a band of width $K \in [5,10]$ pixels along a real gland border. These values of $K$ were found to produce optimal and equivalent quantitative results, measured by the F1 score and the object-Dice index. The output of these networks was used to better calibrate the final prediction.

In the post-processing step, a simple method was applied to clean noise and fill holes in the structures. Thresholding was applied to remove spurious structures with diameter smaller than a certain epsilon. Filling-hole criteria based on diameter size was also used.

Using the initial class discrimination (benign and malignant), a simple binary classifier constructed from a convolutional neural network with 2 convolutional and 1 fully connected layers was trained. This binary classifier used the raw image pixels as input. The output of the classifier was used together with the border networks and the post-processing method to apply a different set of parameters/thresholds depending on the predicted class. The hyperparameters for the entire pipeline, including post-processing and border networks, were obtained through cross-validation.

For this method, the team submitted 3 entries. **ExB 1** was a two-path network including both the border network for detecting borders of glands and the binary classification to differentiate between the post-processing parameters. **ExB 2** was similar to ExB 1 without the use of the border network. **ExB 3** used a two-path network without any post-processing.

## Image Analysis Lab Uni Freiburg[^12]

The authors applied a u-shaped deep convolutional network "u-net"[^13] for the segmentation. The input was the raw RGB image and the output was a binary segmentation map (glands and background). The network consisted of an analysis-path constructed from a sequence of convolutional layers and max-pooling layers, followed by a synthesis path with a sequence of up-convolutional layers and convolutional layers, resulting in 23 layers in total. Additional shortcut-connections propagated the feature maps at all detail levels from the analysis to the synthesis path. The network was trained from scratch in an end-to-end fashion with only the images and ground truth segmentation maps provided by the challenge organizers. To teach the network the desired invariances and to avoid overfitting, the training data were augmented with randomly transformed images and the correspondingly transformed segmentation maps. The applied transformations were random elastic deformations, rotations, shifts, flips, and blurs. The color transformations were random multiplications applied in the HSV color space. To avoid accidentally joining touching objects, a high pixel-wise loss weight was introduced for pixels in thin gaps between objects in the training dataset (see ). The exact same u-net layout with the same hyperparameters as in was used for the challenge. The only difference were more training iterations and a slower decay of the learning rate.

The team submitted two entries. The first entry **Freiburg1** was a connected component labelling applied to the raw network output. The second entry **Freiburg2** post-processed the segmentation maps with morphological hole-filling and deletion of segments smaller than 1000 pixels.

## LIB[^14]

Intestinal glands were divided according to their appearance into three categories: hollow, bounded, and crowded. A hollow gland was composed of lumen and goblet cells and it could be a hole in the tissue surface. A bounded gland had the same composition, but in addition, it was surrounded by a thick epithelial layer. A crowded gland was composed of bunches of epithelial cells clustered together and it might have shown necrotic debris.

The tissue was first classified into one of the above classes before beginning the segmentation. The classification relied on the characterization of the spatial distribution of cells and the topology of the tissue. Therefore, a closing map was generated with a cumulative sum of morphological closing by a disk of increasing radius (1 to 40 pixels) on the binary image of nuclear objects, which were segmented by the $k$-means algorithm in the RGB colour space. The topological features were calculated from a normalized closing map in MSER fashion (Maximally Stable Extremal Region, ) as the number of regions below three different thresholds (25%, 50% and 62.5%) and above one threshold (90%), their sizes and the mean of their corresponding values in the closing map. The first three thresholds characterized the holes and the fourth one characterized the thickness of nuclear objects. After classifying the tissue with a Naive Bayes classifier trained on these features, a specific segmentation algorithm was applied.

Three segmentation algorithms were presented, one for each category. Hollow glands were delineated by morphological dilation on regions below 50%. Bounded gland candidates were first detected as hollow glands, then the thickness of nuclear objects surrounding the region was evaluated by generating a girth map and a solidity map , then after classifying nuclear objects, the epithelial layer was added or the candidate was removed. Crowded glands were identified as populous regions (regions above 90%), and then morphological filtering was applied for refinement. The team submitted a single entry labeled as **LIB** for evaluation.

## vision4GlaS[^15]

Given an H&E-stained RGB histopathological section, the gland segmentation method was based on a pixel-wise classification and an active contour model, and it proceeded in three steps . In a first preprocessing step the image was rescaled to half the spatial resolution, and color deconvolution separated the stained tissue components. The red channel of the deconvolved RGB image represented the tissue structure best and was therefore considered for further processing. Next, two convolutional neural networks (CNNs) of seven layers each were trained for pixel-wise classification on a set of image patches. Each network was trained with ReLU nonlinearities, and stochastic gradient descent with momentum, weight decay, and dropout regularization to minimize a negative log-likelihood loss function. The first CNN, called Object-Net, was trained to distinguish four classes: (i) benign background, (ii) benign gland, (iii) malignant background, and (iv) malignant gland. For each image patch the probability distribution over the class labels was predicted, using a softmax function. The Object-Net consisted of three convolutional layers followed by max-pooling, a final convolutional layer and three fully connected layers. The second – architecturally similar – CNN called Separator-Net, learned to predict pixels of gland-separating structures in a binary classification task. Ground truth was generated by manually labeling image locations, close to two or more gland borders, as gland-separating structures. In the final step the segmentation result was obtained by combining the outputs of the two CNNs. Predictions for benign and malignant glands were merged, and predictions of gland-separating structures were subtracted to emphasize the foreground probabilities. Background classes were handled similarly. Using these refined foreground and background maps, a figure-ground segmentation based on weighted total variation was employed to find a globally optimal solution. This approach optimized a geodesic active contour energy, which minimized contour length while adhering to the refined CNN predictions . The team submitted a single entry, referred to as **vision4GlaS**.

# Results and Discussion

## Summary of the Methods

The methods described above take one of the following two approaches to segmentation: (a) they start by identifying pixels corresponding to glands which are then grouped together to form separated, spatially coherent objects; (b) they begin with candidate objects that are then classified as glands or non-glands. All methods that are based on CNNs (CUMedVision, CVML, ExB, Freiburg, and vision4GlaS) follow the former approach. CVML, ExB, and vision4GlaS built CNN classifiers that assign a gland-related or non-gland-related label to every pixel in an image, by taking patch(es) centered at the pixel as input. ExB, in particular, use multi-path networks into which patches at different sizes are fed, in order to capture contextual information at multiple scales. CUMedVision and Freiburg, on the other hand, base their pixel classifier on a fully convolutional network architecture , allowing simultaneous pixel-wise label assignment at multiple pixel locations. To separate gland-related pixels into individual objects, CVML and vision4GlaS deploy contour based approaches. ExB trains additional networks for glandular boundary, while CUMedVision and Freiburg explicitly include terms for boundary in the training loss function of their networks. The only method that follows the latter approach for object segmentation is LIB. In this method, candidate objects forming part of a gland (i.e., lumen, epithelial boundary) are first identified, and then classified into different types, followed by the final step of segmentation.

A variety of data transformation and augmentation were employed to deal with variation within the data. In order to counter the effect of stain variation, CVML and ExB performed transformations of the RGB color channels, vision4GlaS used a stain deconvolution technique to obtain only the basophilic channel in their preprocessing step. By contrast, Freiburg tackled the issue of stain variability through data augmentation, which implicitly forces the networks to be robust to stain variation to some extent. As is common among methods using CNNs, spatial transformations, such as affine transformations (e.g. translation, rotation, flip), elastic deformations (e.g. pincushion and barrel distortions), and blurring, were also used in the data augmentation to teach the network to learn features that are spatially invariant. The other benefit of data augmentation is it provides, to some extent, avoidance of over-fitting.

ExB, LIB, and vision4GlaS incorporated histologic grades of glands in their segmentation approach. In ExB, procedures and/or parameter values used in boundary detection and post-precessing steps were different, subject to the predicted histologic grade of an image. vision4GlaS classified pixels based on histological information. Although not explicit, LIB categorized candidate objects forming glands according to their appearance, related to histologic grades, before treating them in different ways.

As a post-processing step, many segmentation algorithms employed simple criteria and/or a sequence of morphological operations to improve their segmentation results. A common treatment was to eliminate small spurious segmented objects. Imperfections in pixel labelling can result in the appearance of one or more holes in the middle of an object. Filling such holes is often necessary. In addition to these operations, CVML performed morphological operations to separate accidentally joined objects.

## Evaluation Results

```latex
\resizebox{\columnwidth}{!}{
\begin{tabular}{|c|c|c|c|c|c|c|c|c|c|c|c|c|c|}
\hline
\multirow{3}{*}{Method} & \multicolumn{4}{c|}{$\mathrm{F1score}$}                             & \multicolumn{4}{c|}{$\mathrm{Dice}_\mathrm{obj}$}                                 & \multicolumn{4}{c|}{$\mathrm{H}_\mathrm{obj}$}                            & \multirow{3}{*}{Rank Sum} \\ \cline{2-13}
                        & \multicolumn{2}{c|}{Part A} & \multicolumn{2}{c|}{Part B} & \multicolumn{2}{c|}{Part A} & \multicolumn{2}{c|}{Part B} & \multicolumn{2}{c|}{Part A} & \multicolumn{2}{c|}{Part B} &                           \\ \cline{2-13}
                        & Score         & Rank        & Score         & Rank        & Score         & Rank        & Score         & Rank        & Score          & Rank       & Score          & Rank       &                           \\ \hline
CUMedVision2            & 0.912         & 1           & 0.716         & 3           & 0.897         & 1           & 0.781         & 5           & 45.418         & 1          & 160.347        & 6          & 17                        \\ \hline
ExB1                    & 0.891         & 4           & 0.703         & 4           & 0.882         & 4           & 0.786         & 2           & 57.413         & 6          & 145.575        & 1          & 21                        \\ \hline
ExB3                    & 0.896         & 2           & 0.719         & 2           & 0.886         & 2           & 0.765         & 6           & 57.350         & 5          & 159.873        & 5          & 22                        \\ \hline
Freiburg2               & 0.870         & 5           & 0.695         & 5           & 0.876         & 5           & 0.786         & 3           & 57.093         & 3          & 148.463        & 3          & 24                        \\ \hline
CUMedVision1            & 0.868         & 6           & 0.769         & 1           & 0.867         & 7           & 0.800         & 1           & 74.596         & 7          & 153.646        & 4          & 26                        \\ \hline
ExB2                    & 0.892         & 3           & 0.686         & 6           & 0.884         & 3           & 0.754         & 7           & 54.785         & 2          & 187.442        & 8          & 29                        \\ \hline
Freiburg1               & 0.834         & 7           & 0.605         & 7           & 0.875         & 6           & 0.783         & 4           & 57.194         & 4          & 146.607        & 2          & 30                        \\ \hline
CVML                    & 0.652         & 9           & 0.541         & 8           & 0.644         & 10          & 0.654         & 8           & 155.433        & 10         & 176.244        & 7          & 52                        \\ \hline
LIB                     & 0.777         & 8           & 0.306         & 10          & 0.781         & 8           & 0.617         & 9           & 112.706        & 9          & 190.447        & 9          & 53                        \\ \hline
vision4GlaS             & 0.635         & 10          & 0.527         & 9           & 0.737         & 9           & 0.610         & 10          & 107.491        & 8          & 210.105        & 10         & 56                        \\ \hline
\end{tabular}}
```

Table summarizes the overall evaluation scores and ranks achieved by each entry from each test part. We list the entries according to the order of their rank sum, which indicates the overall performance across evaluation measures and tasks of the entries. The lower the rank sum, the more favorable the performance. The top three entries according to the overall rank sum in descending order are CUMedVision2, ExB1, and ExB3. However, if rank sum is considered with respect to the test part, the three best entries are CUMedVision2, ExB2, and ExB3 for part A; whereas in part B, CUMedVision1, ExB1, and Freiburg2 come at the top. A summary of the ranking results from the competition can be found in . Some segmentation results and their corresponding evaluation scores are illustrated in Figure to give a better idea of how the evaluation scores correlate with the quality of the segmentation.

## Additional Experiments

```latex
\resizebox{0.9\columnwidth}{!}{
\begin{tabular}{|c|c|c|c|c|c|c|}
\hline
\multirow{3}{*}{Entry} & \multicolumn{6}{c|}{Final Ranking}                                                                            \\ \cline{2-7}
                       & \multicolumn{3}{c|}{$\mathrm{F1score} + \mathrm{Dice}_\mathrm{obj}$+$\mathrm{H}_\mathrm{obj}$} & \multicolumn{3}{c|}{$\mathrm{F1score} + \mathrm{ARI} + \mathrm{H}_\mathrm{obj}$} \\ \cline{2-7}
                       & Overall         & Benign         & Malignant         & Overall        & Benign        & Malignant       \\ \hline
CUMedVision1           & 7               & 7              & 3                 & 4              & 6             & 3               \\ \hline
CUMedVision2           & 1               & 1              & 1                 & 1              & 2             & 2               \\ \hline
CVML                   & 10              & 10             & 10                & 10             & 10            & 10              \\ \hline
ExB1                   & 2               & 6              & 2                 & 2              & 7             & 1               \\ \hline
ExB2                   & 6               & 3              & 7                 & 7              & 1             & 7               \\ \hline
ExB3                   & 3               & 5              & 4                 & 3              & 3             & 4               \\ \hline
Freiburg1              & 4               & 4              & 6                 & 6              & 5             & 6               \\ \hline
Freiburg2              & 5               & 2              & 5                 & 5              & 4             & 5               \\ \hline
LIB                    & 8               & 8              & 9                 & 8              & 8             & 9               \\ \hline
vision4GlaS            & 9               & 9              & 8                 & 9              & 9             & 8               \\ \hline
\end{tabular}
}
```

In the challenge, the split of the test data into two parts – Part A (60 images) for off-site test and Part B (20 images) for on-site test – to some extent introduces bias into the performance evaluation of the segmentation algorithms due to equal weight given to performance on the two test parts. The algorithms that perform particularly well on Test Part B would therefore get a better evaluation score even though they may not have performed as well on Test Part A, where the majority of the test dataset is to be found. In addition, the imbalance between the benign and malignant classes in Test Part B, only 4 benign (20%) and 16 malignant (80%) images, would also favor algorithms that perform well on the malignant class. In order to alleviate these issues, we merged the two test parts and re-evaluated the performance of all the entries. In addition, as suggested by one of the participating teams, the adjusted Rand index is included as another performance measurement for segmentation.

The evaluation scores calculated from the combined two test parts are presented as bar chart in Figure . The final rankings based on the rank sums of evaluation scores calculated from the combined two test parts are reported in Table . Here, two set of rank sums are considered: one calculated according to the criteria of the competition (i.e., $\mathrm{F1score} + \mathrm{Dice}_\mathrm{obj} + \mathrm{H}_\mathrm{obj}$), and the other where the adjusted Rand index is used instead of the object-level Dice index to evaluate segmentation accuracy (i.e., $\mathrm{F1score} + \mathrm{ARI} + \mathrm{H}_\mathrm{obj}$). For both sets of rank sums, the new ranking orders are largely similar to those reported in Section , with a few swaps in the order, while the top three entries remaining the same, namely CUMedVision2, ExB1, ExB3.

The main factors that negatively affect the performance of the methods are a number of challenges presented by the dataset. Firstly, large white empty areas corresponding to the lumen of the gastrointestinal tract which are not in the interior of intestinal glands can easily confuse the segmentation algorithms (Figure ). Secondly, characteristics of non-glandular tissue can sometimes resemble that of the glandular tissue. For instance, connective tissue in muscularis mucosa or sub-mucosa layers of the colon is stained white and pinkish and has less dense nuclei, thus resembling the inner part of glands (Figure ). In the case where there is less stain contrast between nuclei and cytoplasm due to elevated levels of Hematoxylin stain, non-glandular tissue with dense nuclei can look similar to malignant epithelial tissue (Figure ). Thirdly, small glandular objects are blended into the surrounding tissue and can be easily mis-detected (Figure ). A careful inspection of the segmentation results generated by each entry showed that methods by CUMedVision, ExB, and Freiburg better avoid over-segmentation or under-segmentation when facing the above-mentioned pitfalls.

The performance of each entry with respect to the histologic grade of cancer was also examined. Their evaluation scores based on benign and malignant samples are reported in the second and the third rows of Figure respectively, and the ranking orders derived from the rank sums of the scores are shown in Table . Based on these results, one can get a better contrast between the performance of the entries that enforce border separation and those that do not. By applying a predicted border mask to separate clumped segmented objects, CUMedVision2 performs better than CUMedVision1, which tends to produce segmentation results that merge neighboring glands together, in both benign and malignant cases. Similarly, ExB1 is able to segment malignant glands better than ExB2 and ExB3 that do not utilize border separation. However, this can have an adverse effect if the algorithm already yields segmentation results that separate individual objects well, such as in the case of ExB1 which under-segments benign glandular objects as compared to its counterparts ExB2 and ExB3.

## General Discussion

The objectives of this challenge were to raise the research community's awareness of the existence of the intestinal gland segmentation problem in routine stained histology images, and at the same time to provide a platform for a standardized comparison of the performance of automatic and semi-automatic algorithms. The challenge attracted a lot of attention from researchers, as can be seen from the number of registered teams/individuals and the number of submissions at each stage of the competition. Interestingly, some of the teams had no experience in working with histology images before. We would like to emphasize that finding the best performing approach is not the main objective of the competition, but rather pushing the boundaries of the-state-of-the-art approaches. Already, we have seen quite interesting developments from many participating teams and the leading algorithms have produced excellent results, both qualitatively and quantitatively.

As noted in the Introduction, morphometric analysis of the appearance of cells and tissues, especially those forming glands from which tumors originate, is one of the key components towards precision medicine, and segmentation is the first step to attain morphological information. Some may have argued that there is no need to perform segmentation, but instead, to follow conventional pattern recognition approaches by extracting mathematical features which normally capture local and/or global tissue architecture and then identifying features that are most suited to the objective of the study. It is true that there are a number of successful works that follow such an approach . However, because these extracted features are often physically less interpretable in the eyes of practitioners, it is difficult to adopt such an approach in clinical settings. On the other hand, the appearance of glands such as size and shape obtained through segmentation is easy to interpret. Segmentation also helps to localize other type of information (e.g., texture, spatial arrangement of cells) that is specific to the glandular areas.

Even though the dataset used in the challenge included images of different histologic grades taken from several patients, it lacked other aspects. First of all, inter-observer variability was not taken into account as the ground truth was generated by a single expert. This is because the intricate and arduous nature of the problem makes it difficult to find several volunteer experts to perform manual segmentation. Considerable experience is required in order to delineate boundaries of malignant glands, which are not so well-defined as those of the benign ones. Moreover, a single image can contain a large number of glands to be segmented, making the task very laborious. Secondly, digitization variability was also not considered in this dataset. It is, in fact, very important to evaluate the robustness of algorithms when the data are scanned by different instruments. As whole-slide scanners are becoming increasingly available, this type of real-world problem should be expected.

The choice of evaluation measures would also affect the comparative results. In this challenge, we emphasized object segmentation and accordingly defined the object-level Dice index and the object-level Hausdorff distance to measure segmentation accuracy at the object level rather than at the pixel level. Nonetheless, it has been suggested that these measures are too strict, as they put a severe penalty on mismatch of the objects. One could replace these measures by less conservative ones, for example, adjusted Rand index or a topology preserving warping error for a volume-based metric and elastic distance for a boundary-based metric. For this reason, we included adjusted rand index as an alternative to object-level Dice index in Section . As we have already pointed out, this results in only a minor change in the ranking order of the entries. Another aspect that was not explicitly included in the evaluation was execution times. Nevertheless, all the algorithms were capable of completing the segmentation task on the on-site test data (Part B) in the given amount of time with or without limitation of resources. Time efficiency is required to process large scale data, such as whole-slide images, whose volume is growing by the day as slides are routinely scanned. Still, in medical practice, accuracy is far more important than speed.

It is worth noting that the used evaluation metrics used here are clinically relevant. As mentioned in the Introduction, morphology of intestinal glands is the key criterion for colorectal cancer grading. This includes shape, size, and formation of the glands. Thus, in terms of clinical relevance, the object-Hausdorff distance is used in accessing the shape similarity between the segmentation results and the ground truth. The object-Dice index is used in assessing the closeness between the volume of the segmentation results and that of the ground truth, which is important in estimating the size of individual glands. Although not directly clinically relevant, F1 score is important in assessing the accuracy of the identified glands. Since the morphological assessment is done on the basis of tissue slide including several thousands of glands, an algorithm with high value of F1 score is more preferable as it can detect a larger number of glands.

Gland segmentation algorithms presented here are not ready for deployment into clinic in their present form. Although some of the top algorithms produce good segmentation results for the contest dataset and will probably fare well in the real world, there needs to be a large-scale validation involving data from multiple centers annotated by multiple pathologists before any of these algorithms can be deployed in a diagnostic application.

The challenge is now completed, but the dataset will remain available for research purposes so as to continually attract newcomers to the problem and to encourage development of state-of-the-art methods. Extension of the dataset to address inter-observer and inter-scanner variability seems to be the most achievable aim in the near future. Beyond the scope of segmentation, there lie various extremely interesting future research directions. Previous studies have shown the strong association between the survival of colorectal cancer patients and tumor-related characteristics, including lymphocytic infiltration , desmoplasia , tumor budding , and necrosis . A systematic analysis of these characteristics with the help of gland segmentation as part of automatic image analysis framework could lead to a better understanding of the relevant cancer biology as well as bring precision and accuracy into assessment and prediction of the outcome of the cancer.

# Conclusions

This paper presented a summary of the Gland Segmentation in Colon Histology Images (GlaS) Challenge Contest which was held in conjunction with the 18th International Conference on Medical Image Computing and Computer Assisted Interventions (MICCAI'2015). The goal of the challenge was to bring together researchers interested in the gland segmentation problem, to validate the performance of their existing or newly invented algorithms on the same standard dataset. In the final round, the total number of submitted entries for evaluation was 19, and we presented here in this paper 10 of the leading entries. The dataset used in the challenge has been made publicly available and can be accessed at the challenge website (<http://www.warwick.ac.uk/bialab/GlasContest/>). Those who are interested in developing or improving their own approaches are encouraged to use this dataset for quantitative evaluation.

# Acknowledgements

This paper was made possible by NPRP grant number NPRP5-1345-1-228 from the Qatar National Research Fund (a member of Qatar Foundation). The statements made herein are solely the responsibility of the authors. Korsuk Sirinukunwattana acknowledges the partial financial support provided by the Department of Computer Science, University of Warwick, UK. CUMedVision team acknowledges Hong Kong RGC General Research Fund (Project No. CUHK 412513). The work of Olaf Ronneberger was supported by the Excellence Initiative of the German Federal and State Governments (EXC 294). Philipp Kainz was supported by the Excellence Grant 2014 of the Federation of Austrian Industries (IV), and Martin Urschler acknowledges funding by the Austrian Science Fund (FWF): P 28078-N33.

The authors thank Professor David Epstein for his extensive help with the wording and mathematical notation in this paper, and Nicholas Trahearn for his help with the wording.

# The Complete Contest Results

A summary of the ranking results from the contest is given in Figure .

[^1]: email@example.com

[^2]: email@example.com

[^3]: corresponding author

[^4]: <http://www.warwick.ac.uk/bialab/GlaScontest>

[^5]: The T in TNM cancer staging refers to the spread of the primary tumour. In colorectal cancer, stage T3 means the tumour has grown into the outer lining of the bowel wall, whereas stage T4 means the tumour has grown through the outer lining of the bowel wall. The cancer stage is different from the tumour histologic grade, as the latter indicates the aggressiveness of the tumour.

[^6]: <http://www.warwick.ac.uk/bialab/GlaScontest>

[^7]: Department of Computer Science and Engineering, The Chinese University of Hong Kong.

[^8]: <http://host.robots.ox.ac.uk:8080/pascal/VOC/voc2012/index.html>

[^9]: More details will be available at: <http://www.cse.cuhk.edu.hk/~hchen/research/2015miccai_gland.html>

[^10]: School of Engineering, University of Central Lancashire, Preston, UK.

[^11]: ExB Research and Development.

[^12]: Computer Science Department and BIOSS Centre for Biological Signalling Studies, University of Freiburg, Germany.

[^13]: The implementation of the u-net is freely available at <http://lmb.informatik.uni-freiburg.de/people/ronneber/u-net/>.

[^14]: Sorbonne Universités, UPMC Univ Paris 06, CNRS, INSERM, Biomedical Imaging Laboratory (LIB), Paris, France.

[^15]: Institute of Biophysics, Center for Physiological Medicine, Medical University of Graz, Graz, Austria; Institute of Neuroinformatics, University of Zurich and ETH Zurich, Zurich, Switzerland; Institute for Computer Graphics and Vision, BioTechMed, Graz University of Technology, Graz, Austria; Ludwig Boltzmann Institute for Clinical Forensic Imaging, Graz, Austria.


abstract: We present the highest resolution VLBI imaging observations of Sgr A\* made at both 7 and 3.5 mm. These data reveal wavelength-dependent intrinsic sizes with an intrinsic emitting region at 3.5 mm of about 1 AU (at a distance of 8 kpc to the Galactic Center). When combined with the lower limit on the mass of Sgr A\*, these size measurements provide strong evidence that Sgr A\* is a super-massive black hole. We also detected a structural variation which results in an intrinsically symmetrical structure that increases in its intrinsic size by more than 25% at 7 mm.
address: Shanghai Astronomical Observatory, 80 Nandan Road, Shanghai 200030, China
author: Zhi-Qiang Shen[^1]
title: High-resolution Millimeter-VLBI Imaging of Sgr A\*

# Introduction

Sgr A\*, the extremely compact non-thermal radio source at the galactic center, is a best candidate for a single super-massive black hole (SMBH) from both the observational studies and the theoretical models (cf. ). The determination of the orbital motions of those early-type stars within the vicinity of Sgr A\* reveal a central dark mass about 4$\times10^6{\rm\
M_\odot}$ within a radius of about 45 AU . On the other hand, the upper limit to the intrinsic proper motion of Sgr A\* itself sets a lower limit of 0.4$\times10^6{\rm\ M_\odot}$ to the mass directly associated with Sgr A\* . Now we can say that we have known the mass of Sgr A\* to a great extent, the main uncertainty being the exact distance to the galactic center. But how big it is, and what it looks like, are different questions.

VLBI observations can provide the highest angular resolution achievable with any astronomical instruments . At a distance of 8 kc to Sgr A\* at the galactic center , an angular separation of 1 mill-arcsecond (mas) corresponds to a linear size of 8 AU. The Schwarzschild radius ($\rm R_{sc}$) of a 4$\times10^6{\rm\ M_\odot}$ SMBH is 1.2$\times$`<!-- -->`{=html}10$^{12}$ cm, or 0.08 AU or 10 micro-arcsecond ($\mu$as) in angular size. Indeed, Sgr A\* is the closest SMBH candidate with the largest angular size of its Schwarzschild radius on the sky, and thus one of the best prime targets for the VLBI observational study.

# Millimeter-VLBI of Sgr A\*

Attempts to measure the Sgr A\* structure with high-resolution VLBI observations at centimeter (cm) wavelength have suffered from the angular broadening caused by the diffractive scattering by the ionized interstellar medium, which dominates the resultant images with a $\lambda^2$-dependence apparent size, that is, $\Theta_{\rm
obs}=\Theta_{\rm obs}^{\rm 1 cm}\lambda^2$, where $\Theta_{\rm
obs}$ is the observed full-width at half-maximum (FWHM) in mas at a wavelength $\lambda$ in cm, and equals $\Theta_{\rm obs}^{\rm 1
cm}$ at 1 cm. This tells us that the cm radio emission of Sgr A\* comes from a region that extends to observers a much smaller angle than the scattering angular size. However, following the $\lambda^2$-dependent scattering law, the scattering angle decreases very fast with the observing wavelength. So it can be expected that VLBI observations at millimeter (mm) wavelength, where the pure scattering angular size could become small enough to be comparable to the intrinsic (finite) size of Sgr A\*, would show deviations of the observed apparent size from the scattering law.

Unlike the cm-VLBI observations, the mm-VLBI observations are severely limited by the atmosphere due to the fact that Sgr A\* has a southerly declination ($\sim- 30^\circ$) and most existing VLBI antennas are located in the Northern Hemisphere. Therefore, most of the observational data of Sgr A\* were taken at low elevation angles ($10^\circ - 20^\circ$) where atmospheric effects are substantial. Furthermore, the short and variable coherence time at mm wavelength, combined with the compromised sensitivity due to the high system temperature and the lower antenna efficiency of mm VLBI antennas, seriously limits the high signal-to-noise-ratio (SNR) detections of mm-VLBI observations of Sgr A\*. As a consequence, there are large uncertainties in the results obtained from the conventional imaging process, i.e. the self-calibration technique for VLBI imaging, whose biggest drawback is non-uniqueness when the SNR is poor.

## Revision of Scattering Law

To circumvent the large uncertainty in the commonly used amplitude calibration of VLBA observations of Sgr A\*, we have since 2001 developed a model fitting program in which the amplitude closure relation is applied to minimize the calibration errors and thus to improve the accuracy of the measurements . We then applied this procedure to refine the apparent size measurements of the first near-simultaneous VLBA+Y1 data observed at five wavelengths (6, 3.6, 2, 1.35 cm, and 7 mm). By performing the weighted least-squares fits to these sizes at different wavelengths, we conclude that the best-fit two-dimensional scattering structure is $$\Theta_{\rm sca}^{\rm major} = (1.39 \pm 0.02)\lambda^2 ~,$$ and $$\Theta_{\rm sca}^{\rm minor} = (0.69\pm0.06)\lambda^2 ~,$$ with a position angle of 80$^\circ$. Here, $\Theta_{\rm sca}^{\rm
major}$ and $\Theta_{\rm sca}^{\rm minor}$ are FWHM of major and minor axes in mas. This newly revised wavelength-dependent two-dimensional scattering structure is consistent with that from directly fitting to the closure amplitudes of somewhat different databases . Compared with previous one (cf. ), new scattering law shows an even smaller scattering angular size along the East-West major axis direction.

## Detections of Intrinsic Size

On November 3, 2002, we successfully carried out the first VLBA imaging observation of Sgr A\* at its shortest wavelength of 3.5 mm . Compared to the past 3.5 mm VLBI observations with the CMVA , our VLBA observation, with its dynamic scheduling, the highest possible recording rate and more frequent pointing calibration, has for the first time determined its elliptical structure in the radio emission at 3.5 mm, consistent with the morphology seen at longer wavelengths. It should be pointed out that dynamic observing is crucial to minimize the atmospheric effects in the mm-VLBI observation of Sgr A\*. The exact observing date is not fixed, but depends on the weather condition at most antenna sites which usually are separated by a few thousand kilometers. While observing, a frequent antenna pointing check observations of compact and strong SiO maser sources, roughly in every 10 minutes, is necessary to maintain the best reference (offset) pointing and calibration.

The best fitted apparent structure from November 2002 VLBA observations is 0.21$^{+0.02}_{-0.01}$ mas by 0.13$^{+0.05}_{-0.13}$ mas with a position angle 79$^{+12}_{-33}$ degrees. The major axis size is well determined while the fitted size of the minor axis has a quite big error mainly due to the poor resolution along the North-South direction. By subtracting in quadrature the scattering angle from Eq. (1), we obtain, for the first time, an intrinsic size of 0.126$\pm$`<!-- -->`{=html}0.017 mas along the major axis at 3.5 mm. In September 2003, we performed the second 3.5 mm VLBA imaging observation of Sgr A\*, which confirms the results in November 2002 (Shen et al. 2006 in preparation).

Similarly, two 7 mm VLBA+GBT (Green Bank Telescope) observations with the highest recording rate were dynamically scheduled in March 2004 with the resultant averaged sizes of the major and minor axes of 0.724$\pm$`<!-- -->`{=html}0.001 mas and 0.384$\pm$`<!-- -->`{=html}0.013 mas, respectively with position angle 80.6$^{+0.5}_{-0.6}$ degrees. The inclusion of GBT can improve the resolution in North-South direction by a factor of 3 , and actually this is the first VLBI observations to make use of the GBT at 7 mm. The results are consistent with the previous ones, but with much better accuracies. A significant deviation of measured size from the scattering angle along the major axis indicates an intrinsic size of 0.268$\pm$`<!-- -->`{=html}0.010 mas for the major axis , consistent with the previously reported detection of intrinsic size at 7 mm .

## Temporal Variation in the Structure of Sgr A\*

On May 31, 1999, Sgr A\* was observed simultaneously at three 7 mm bands (39, 43 and 45 GHz). It shows a larger deviation in the apparent source size from scattering angle than that observed a week ago on May 23, 1999 . Such a deviation is significant at about 3$\sigma$ along the minor axis direction. Thus, for the first time we can estimate an intrinsic size along the minor axis of 0.359$\pm$`<!-- -->`{=html}0.095 mas. The derived intrinsic size for the major axis is 0.334$\pm$`<!-- -->`{=html}0.042 mas. Therefore, within the error bars, the intrinsic structure detected at 7 mm on May 31, 1999 could be treated as a circular. Compared with the previously detected major axis size of 0.268 mas, such an intrinsic structure has increased in its size by at least 25% at 7 mm.

# Discussion

We have sampled a zone of the SMBH closer to the event horizon than ever before, by detecting the intrinsic size of Sgr A\* to be only 1 AU, or 12.6 R$_{\rm sc}$. Assuming an intrinsically spherical structure of Sgr A\* and using the lower limit to the mass of Sgr A\* of 4$\times10^5{\rm\ M_\odot}$ from the absolute proper motion work , we can easily derive a lower limit to the dark mass density of Sgr A\* of 6.5$\times$`<!-- -->`{=html}10$^{21}{\rm\
M_\odot}~{\rm pc}^{-3}$. This is the highest mass density ever measured in any SMBH candidates, and thus strongly supports the SMBH nature of Sgr A\*.

It is intriguing that the detected intrinsic size at 3.5 mm is about twice the diameter of the shadow caused by the strong gravitational bending of light rays . Regardless of the exact emission model, the characteristic diameter of shadow is always about 5R$_{\rm sc}$, which is nearly 50 $\mu$as for Sgr A\*. So, Sgr A\* will be the most important target for the future sub-mm VLBI experiment to test the general relativity in the strong field regime. Such kind of shadow, if confirmed, would be the most direct evidence for the existence of SMBH. The success of earlier single baseline 1.3 mm VLBI experiments has already demonstrated the feasibility of capturing an image of the shadow around the edge of Sgr A\* at sub-mm wavelengths in the near future.

From the two well determined intrinsic size at 7 and 3.5 mm , we can derive a $\lambda^\beta$-dependence of the intrinsic size with $\beta=1.09^{+0.34}_{-0.32}$ using the two-point fit. This intrinsic size versus wavelength relation is consistent with the two lower limits of 0.02 and 0.008 mas at 1.3 and 0.8 mm, respectively, from the absence of refractive scintillation .

Such a wavelength dependence of the intrinsic major axis size provides a strong constraint on emission and accretion models for Sgr A\*. It explicitly rules out explanations other than those models with stratified structure. It has been shown that the radiatively inefficient accretion flow (RIAF) model, after taking into account the interstellar scattering, can predict the observed sizes at both 7 and 3.5 mm quite well . The extrapolated size of emitting region at 1 mm will reach the last stable orbit (LSO) radius of 3R$_{\rm sc}$ for a non-rotating SMBH. For a rotating SMBH, the radius of LSO could be only 0.5R$_{\rm sc}$. Therefore, a break in the wavelength-dependent intrinsic size is inevitable with the decreasing wavelength, which can be used to constrain the spin of SMBH.

The radio flux density of Sgr A\* is known to be variable at all the observable wavelengths on time scales of days to months. The variability appears to be more pronounced at shorter wavelengths with a relatively large amplitude fluctuation. For example, an intra-day variability (IDV) in the 2 mm flux density of Sgr A\* has been found by the Nobeyama Millimeter Array (NMA) observations, indicating a very compact emitting region . The correlation between flux density at 7 mm and spectral index suggests that the variation at the short radio wavelengths is intrinsic rather than due to the interstellar scintillation . As such, it seems inevitable that the variability in the radio flux density of Sgr A\* would be accompanied by the structural change. Our simulation seems to suggest that the detected structural variability on May 31, 1999 could be related to an outburst/flare occurred at about 50R$_{\rm sc}$ from the central SMBH (Shen et al. in preparation).

# Conclusions

We have revised the scattering law, which gives a smaller scattering size along the East-West major axis direction. With the dynamic scheduling, we successfully carried out two 3.5 mm VLBA imaging observations, which reveal a consistent East-West elongated apparent structure. We also performed two-epoch new 7 mm VLBA+GBT observations with a better sensitivity and resolution. These new data show that the intrinsic size has come to play with the scattering effect at both 3.5 and 7 mm. The inferred intrinsic size is about 1 and 2.1 AU at 3.5 and 7 mm, respectively. The derived extraordinarily high (dark) mass density of Sgr A\* strongly argues that Sgr A\* is indeed a super-massive black hole.

This work was supported in part by the National Natural Science Foundation of China (grant 10573029). Z.-Q. Shen acknowledges support by the One-Hundred-Talent Program of the Chinese Academy of Sciences. In addition to the author, people who are involved in the project are P. T. P. Ho (ASIAA), M.-C. Liang (Caltech), K. Y. Lo (NRAO), M. Miyoshi (NAOJ) and J.-H. Zhao (CfA).

# References

[^1]: for collaborators see acknowledgments