---
abstract: 'Medical image annotation typically requires expert knowledge and hence incurs time-consuming and expensive data annotation costs. To reduce this burden, we propose a novel learning scenario, Exemplar Learning (EL), to explore automated learning processes for medical image segmentation from a single annotated image example. This innovative learning task is particularly suitable for medical image segmentation, where all categories of organs can be presented in one single image for annotation vhardy@example.net. To address this challenging EL task, we propose an Exemplar Learning-based Synthesis Net (ELSNet) framework for medical image segmentation that enables innovative exemplar-based data synthesis, pixel-prototype based contrastive embedding learning, and pseudo-label based exploitation of the unlabeled data. Specifically, ELSNet introduces two new modules for image segmentation: an exemplar-guided synthesis module, which enriches and diversifies the training set by synthesizing annotated samples from the given exemplar, and a pixel-prototype based contrastive embedding module, which enhances the discriminative capacity of the base segmentation model via contrastive self-supervised learning. Moreover, we deploy a two-stage process for segmentation model training, which exploits the unlabeled data with predicted pseudo segmentation labels. To evaluate this new learning framework, we conduct extensive experiments on several organ segmentation datasets and present an in-depth analysis. The empirical results show that the proposed exemplar learning framework produces effective segmentation results.'
author:
- |
    **Qing En, Yuhong Guo**\
    Carleton University, Ottawa, Canada\
    `vhardy@example.net, vhardy@example.net`\
bibliography:
- 'references.bib'
title: Exemplar Learning for Medical Image Segmentation
---

Introduction
============

Medical image analysis is becoming increasingly important for clinical diagnosis and surgical planning due to the rapid advancement of medical imaging technologies @duncan2000medical [@tajbakhsh2016convolutional]. Notably, medical image segmentation is one of the critical steps in quantitative medical image analysis, aiming to automatically identify the target region from medical images pixel-by-pixel @sharma2010automated [@pham2000current]. Fully supervised deep neural networks have been demonstrated to yield desirable segmentation results by using large amounts of labeled training data @chen2021transunet [@wang2021mixed], while limited training data often leads to a degraded performance due to over-fitting. However, obtaining abundant annotated medical images at the pixel-level entails substantial labour and financial expenses because the images require the knowledge of clinical experts that is not always available. To reduce the annotation cost, several techniques have been developed as potential solutions to perform medical image segmentation with less annotated data @fei2006one, including semi-supervised segmentation @mittal2019semi [@reiss2021every; @wu2021collaborative] and few-shot segmentation @sung2018learning [@roy2020squeeze].

Although some notable improvements have been achieved, current solutions are still unable to eliminate the labelling conundrum. Most few-shot segmentation methods rely on extensive auxiliary training datasets with exhaustive annotated data to transfer knowledge from the support set to the query set @roy2020squeeze [@tang2021recurrent]. In semi-supervised segmentation, existing methods usually focus on exploiting the consistency property of unlabeled data, but they still require a nontrivial portion of the densely annotated data @wu2021collaborative [@seibold2021reference; @luo2021semi]. Due to the differences in acquisition devices or techniques, the demand for annotated data is often particularly amplified for medical image analysis. Meanwhile, we observe that the images for medical segmentation tasks often contain variations of the same set of organ categories, while a proper example image can cover all the parts for the whole organ category set. Motivated by this observation, we propose a novel learning scenario called Exemplar Learning (EL) to set up the working environment for a new set of medical image segmentation techniques that only require one single expert annotated image. The differences between this new EL setting and the previous semi-supervised and few-shot segmentation settings are illustrated in Fig. \[fig:fig1\].

Recently, Ouyang <span>et al. </span>@ouyang2020self proposed training a self-supervised few-shot segmentation framework using super-pixels and local prototype information. But they still require densely annotated support sets in the testing phase, and their segmentation results lack semantic information. Some methods leverage multiple types of labels or use a small number of labeled images as reference material to match unlabeled images @reiss2021every [@seibold2021reference]. However, our EL scenario is more challenging than those described above because only one annotation image is available in the training stage, and no support sets are provided during testing.

The fundamental challenges for exemplar learning lie in the following two aspects: (1) Data variety is severely deficient, and the number of foreground-background pixels is imbalanced in medical image datasets. Since only one annotated image is available, the model can easily be overfitted to the labeled sample. In addition, the model lacks proper guidance throughout the training process when facing massive background pixels, which can easily result in a considerable drop in performance when noise is encountered. (2) The contrast between the organ and the background is low, and the differences in appearance among organs are barely discernible, leading to a lack of discriminatory ability of the model. Medical images struggle with a scenario where the background is heterogeneous, while the foreground is homogeneous. This phenomenon makes distinguishing the boundaries between organs difficult, leading to over-segmentation issues.

The fact that humans can learn by analogy @bar2007proactive [@ryali2021learning; @zhao2021distilling] motivates us to address the abovementioned challenges by enriching the sample diversity and the discriminability of models.

![image](Fig1.pdf){width="0.82\linewidth" height="48mm"}\

In this paper, we propose *ELSNet*, a novel framework for learning to segment medical images effectively with only one annotated image. ELSNet enriches labeled data diversity by synthesizing training data and enhances the discriminability of the base segmentation model by performing pixel-prototype based contrastive embedding learning. Specifically, given the exemplar image with all of the categories labeled, we first devise an exemplar-guided synthesis module (ESM) to enlarge our training set by taking crops of foreground organs and pasting them with various transformations onto different backgrounds images. This encourages organ representation to be invariant to the background, while enriching the diversity of the sample. Next, we design a pixel-prototype based contrastive embedding module (PCEM) to decompose the organs into distinct and consistent parts by capturing homogeneous components of the same type in a contrastive self-supervised embedding learning. We adopt the contrastive formulation to constrain prototypes to improve the discriminability of the model. This module enables pixels belonging to the same organ to be similar compared to the case where pixels belong to a different organ. Moreover, we deploy a two-stage process for segmentation model training, which exploits the unlabeled data with predicted pseudo segmentation labels. Instead of relying on multiple annotations or online guidance, the proposed framework can perform segmentation with only one annotated image from a synthetic and contrastive perspective. The main contributions of our paper can be summarized as follows:

-   We propose a novel learning scenario, Exemplar Learning, which explores medical image segmentation from a single annotated image.

-   We propose a novel ELSNet framework to segment medical images in the EL scenario by creating exemplar-based data, learning pixel-prototype based contrastive embedding and exploring unlabeled data with pseudo-labels.

-   Experimental results on two medical image segmentation datasets show that the proposed ELSNet can effectively perform the medical semantic segmentation task.

Related work
============

Self-Supervised Learning in Medical Image Analysis
--------------------------------------------------

Recent approaches of self-supervised learning in medical image analysis are proposed by designing pretext-based tasks @tao2020revisiting [@zhu2020rubik; @zhuang2019self; @chen2019self; @zhou2021models; @taleb20203d; @zhou2021generalized; @zhou2021ssmd; @sowrirajan2021moco]. In addition, a collection of generic source models are built for generating powerful application-specific target models through transfer learning from unlabeled 3D medical image data @zhang2020universal [@zhou2019models]. In @chaitanya2020contrastive [@yan2020self], the proposed models can determine the distribution of transformed samples by defining the global and the local contrastive loss. To pretrain segmentation models, PGL @xie2020pgl proposed learning the region-wise local consistency between two views to retain structural information for self-supervised 3D medical image segmentation. The above methods require fine-tuning the network on downstream tasks after self-supervised learning, where different annotation information is required for this process. However, the proposed exemplar learning requires only one sample annotation in the training set without fine-tuning.

Semi-Supervised Medical Image Segmentation
------------------------------------------

Semi-supervised semantic segmentation has received increasing attention to train models by reducing the mask labeling cost @souly2017semi [@mittal2019semi; @zou2020pseudoseg; @ouali2020semi; @sohn2020fixmatch]. The technique has also been applied to the field of medical image segmentation @reiss2021every [@wu2021collaborative; @seibold2021reference; @luo2021semi]. Reiß <span>et al. </span>@reiss2021every proposed a multi-label deep supervision model to supervise low-resolution features and apply it to the multiple medical supervision signals. Wu <span>et al. </span>@wu2021collaborative presented a semi-supervised polyp segmentation model by collaborative and adversarial learning. Moreover, Seibold <span>et al. </span>@seibold2021reference used labeled images as references to generate more accurate pseudo-labels. Luo <span>et al. </span>@luo2021semi proposed a dual-task- consistency semi-supervised framework to address this problem. These methods require multiple annotated samples or scans to achieve this task. In contrast, we propose to solve the medical image segmentation problem in a scenario with only one annotated image, which is more challenging.

Few-Shot Medical Image Segmentation
-----------------------------------

In the last few years, few-shot segmentation (FSS) has been exploited in the medical image domain. @zhao2019data [@mondal2018few; @ouyang2019data; @yu2020foal; @chen2020realistic] Most of these methods require many base categories to be annotated during the training phase and require fine-tuning for unseen classes. SE-Net @roy2020squeeze and RP-Net @tang2021recurrent focus on designing models to segment unseen classes without retraining. Moreover, Sli2Vol @yeung2021sli2vol propagated the 2D image segmentation with an affinity matrix directly to reconstruct the rest of the image in 3D volumes in a self-supervised manner. Ouyang <span>et al. </span>@ouyang2020self generated superpixel-based pseudo-labels and used the adaptive local prototype information for training the self-supervised FSS framework. However, these methods require the support set in the test phase, and the predicted masks do not have semantic information. In contrast, the proposed exemplar learning does not require a support set, and the predicted masks have semantic information.

Proposed Method
===============

In the setting of the exemplar learning, one labeled training image (<span>i.e. </span>Exemplar) and T unlabeled training images are given, and denoted as $\mathcal{D}_{E} = (I_{e}, Y_{e})$ and $\mathcal{D}_{U} = \{(I_{u}^t)\}_{t=1}^{T}$. The input image is defined as $I\in \mathbb{R}^{1*H*W}$, and the label is defined as $Y_{e}\in \{0, 1\}^{K*H*W}$, where $K, H, W$ are the number of categories in the dataset, height and width of the input image, respectively. In this setting, the single exemplar image contains one segmentation instance for each category. We aim to train the proposed ELSNet to generate semantic masks.

![image](Fig2.pdf){width="0.9\linewidth" height="100mm"}\

The overall architecture of the ELSNet is illustrated in Fig. 2. We first create the synthetic dataset from the given exemplar by the exemplar-guided synthesis module, which enriches and diversifies the training set. Moreover, we deploy a two-stage process for training the segmentation model by using the pixel-prototype based contrastive embedding module, which enhances the discriminative capacity of the base segmentation model and exploits the unlabeled data with predicted pseudo segmentation labels. In the first stage, a synthetic segmentation network $\mathcal{N}_{s}$ is trained by the exemplar and the synthetic dataset, which is then used to generate pseudo-labels for unlabeled data. In the second stage, an exemplar learning segmentation network $\mathcal{N}_{e}$ is trained by the exemplar, the synthetic dataset and the unlabeled dataset, which is used for segmentation during testing. Two segmentation networks consist of an encoder $f_{encoder}: \mathbb{R}^{1*H*W} \to \mathbb{R}^{c*h*w}$ and a decoder $f_{decoder}: \mathbb{R}^{c*h*w} \to \mathbb{R}^{K*H*W}$, respectively. $c, h,$ and $w$ are the channel, height, and width of the embedding, which is represented as $X=f_{encoder}(I)$. Subsequently, the learnable parameters of the segmentation networks are updated based on four losses as follows: $$L_{total} = \lambda_{e}L_{e}+\lambda_{s}L_{s}+\lambda_{c}L_{c}+\lambda_{u}L_{u}
,
\label{total losses}$$ where the exemplar loss $L_{e}$ is computed from the exemplar; the synthetic segmentation loss $L_{s}$ is computed from the synthetic dataset (Section \[Exemplar-guided Sample Synthesis Module\]); the prototype contrastive loss $L_{c}$ is designed to make a distinct discrimination between the different organs (Section \[Prototype Contrastive Module\]); the unlabeled segmentation loss $L_{u}$ is computed from the unlabeled dataset; and $\lambda_{e}$, $\lambda_{s}$, $\lambda_{c}$ and $\lambda_{u}$ are the weights of the proposed losses. We describe the details below.

Exemplar-Guided Synthesis Module {#Exemplar-guided Sample Synthesis Module}
--------------------------------

This module aims to synthesize the segmentation instance of each label category from the single exemplar into various backgrounds, thus creating a synthetic training dataset. The diagram of this module is shown in Fig. 2. Given an exemplar, we first obtain the appearance of each organ according to their annotations, transform them to imitate the various organs in medical images, and finally paste the transformed organs onto background images. For this purpose, we define a series of geometric transform operations and intensity transform operations as $\mathcal{T}_{g}$ and $\mathcal{T}_{i}$, respectively. Subsequently, a synthetic sample $I_{s}$ and its label $Y_{s}$ are generated by performing different transformations on $I_{e}$ and $Y_{e}$ as follows: $$\begin{split}
(I_{s}, Y_{s}) = \Omega(\mathcal{T}_{g}(\mathcal{T}_{i}(I_{e})), \mathcal{T}_{g}(Y_{e}), \mathcal{T}_{g}(\mathcal{T}_{i}(I_{b})),
\end{split}
\label{transformation}$$ where $\Omega$ represents the proposed exemplar-guided synthesis operation that copies, transforms and pastes the exemplar onto the selected background image $I_{b}$. We choose images in the dataset that do not contain any organs and black images as background images. Meanwhile, the corresponding label $Y_{s}$ is generated based on the transformed $Y_{e}$. By this means, the synthetic training dataset is created by using Eq. (\[transformation\]) with sufficient variations based on only one exemplar.

To implement the $\Omega$ operation, we first segregate the exemplar $I_{e}$ into different categories of organs as follows: $$I_{e}^k = I_{e} \otimes Y_{e}^k,
\label{exemplar_organ}$$ where $Y_{e}^k$ and $I^k_{e}$ indicate the mask and the exemplar of the *kth* category organ, respectively. Then, for the segmentation instance for each category from the single exemplar, geometric and intensity transformations are applied to imitate scale change, distortion, and lighting variations across the dataset. Notably, we perform the same transformation operation with various parameters for different instances separately to enrich sample diversity. Because the selected background images would not adequately cover all scenarios, we also apply a slight transformation of the background images to further enhance the diversity of the dataset. Finally, we compose the transformed exemplar organs and paste them onto the background images, creating a synthetic sample. As a result, the synthetic dataset $\mathcal{D}_{S} = \{(I_{s}^b, Y_{s}^b)\}_{b=1}^{B}$ is created, where $I_{s}^b\in \mathbb{R}^{1*H*W}$, $Y_{s}^b\in \{0, 1\}^{K*H*W}$ and $B$ is the number of synthetic samples.

In this module, as the basic features of all organs are present in the exemplar, the synthetic dataset allows the construction of training data to train the model in routine ways without extra annotation. Consequently, this module is explicitly designed for medical images, where the synthetic dataset overcomes the limitations of previous work that relies on virtual data that has a gap with the real world @dosovitskiy2015flownet.

![image](Fig3.pdf){width="0.8\linewidth" height="40mm"}\

Pixel-Prototype Based Contrastive Embedding Module {#Prototype Contrastive Module}
--------------------------------------------------

Since the appearance of different organs in a medical image is nearly identical, and the frontiers of the organs are vague, distractions in the background may hinder the segmentation results. To address this problem, we focus on calculating prototypes of different organs and constraining them using a contrast learning paradigm to improve the discriminability of the model.

The PCEM starts by calculating prototypes by efficiently extracting global object representations. As previously stated, $X$ represents the embedding features of the input image, and we use $f_{decoder}$ to generate the prediction mask $\hat{Y}$ as follows: $$\hat{Y}=argmax(softmax(f_{decoder}(X)))
,
\label{eq1}$$ where $softmax$ represents the softmax function and $\hat{Y}\in \mathbb{R}^{K*H*W}$. Global features of images have been demonstrated to be useful for segmentation tasks @zhang2019canet [@zhang2020sg]. Therefore, we leverage global average pooling over the foreground to squeeze the object features into a feature vector, which is seen as the prototypes of the organs. We resize the predicted mask $\hat{Y}\in\mathbb{R}^{K*H*W}$ to the same size as the embedding features $X$ via bilinear interpolation, which is denoted by $\hat{Y}_{x}\in\mathbb{R}^{K*h*w}$. To calculate the prototype for the $kth$ category $v_k$, masked average pooling is adopted to extract a global object representation for the target organs, as follows: $$\mathnormal{v}_{k}=\frac{\sum_{(i,j)}X^{(i,j)}\mathbbm{1}[\hat{Y}_{x}^{(k,i,j)}\neq0]}{\sum_{(i,j)}\mathbbm{1}[\hat{Y}_{x}^{(k,i,j)}\neq0]}
,
\label{SOFG}$$ where $(k,i,j)$ indicates the spatial location index of the $kth$ category and $\mathbbm{1}$ represents the indicator function. In this operation, we apply elementwise multiplication between the resized predicted mask and the embeddings, where features belonging to the background area become zero.

To maximize the similarity of the same organ among different images while simultaneously minimizing the similarity of distinct organs, we propose to exploit contrast formulation in the prototype space to approach this issue. We compute the similarity of prototypes among different images using a batch of $N$ samples (<span>e.g. </span>, a mini-batch) and use this as the element to calculate the score to construct the contrastive relationship. Specifically, for each category k, we randomly select a prototype $v_k^m\in\{v_k^i, i\not= n | v_k^n\}$ from the other $N-1$ images in the current batch as a positive sample. For this purpose, the prototype contrastive loss is defined as follows: $$L_{c}=-\sum_{n=1}^{N}\sum_{k=1}^{K}\log\frac{exp(v_{k}^{n} \cdot v_{k}^{m} / \tau)}{exp(v_{k}^{n} \cdot v_{k}^{m} / \tau) + \sum_{j \neq k}\sum_{i \neq n} exp(v_{k}^{n} \cdot v_{j}^{i} / \tau)}
,
\label{cont_loss}$$ where $\tau$ is the temperature hyper-parameter; $n$ is the index of the images.

The proposed module is designed with two main differences relative to the previous self-supervised approaches, allowing for a more effective response to the issues faced in our experimental scenario. First, we constrain prototypes of various organs that rely on the mask rather than on the entire image, which is directly related to the objective of the segmentation task. Second, instead of augmenting the inputs, we perform contrastive learning across multiple images, based on the fact that the semantic representation of the same organ is invariant across images. Based on these two characteristics, PCEM can encourage the same organ to be similar in other images, though not similar to other organs in this and other images simultaneously. As qualitatively illustrated in Fig. \[fig:fig3\], ESM and PCEM can effectively guide the intermediate embeddings to focus on the target objects.

Two-Stage Training
------------------

The ELSNet is trained in two stages. In the first stage, the exemplar $\mathcal{D}_{E}$ and the synthetic dataset $\mathcal{D}_{S}$ are used as training data to train the $\mathcal{N}_{s}$ by computing three losses as follows: $$L_{s1} = \lambda_{e}L_{e}+\lambda_{s}L_{s}+\lambda_{c}L_{c}
.
\label{stage2loss}$$ Then, we input $\mathcal{D}_{U}$ into $\mathcal{N}_{s}$ to obtain their predicted masks $Y_{u}$ as their pseudo-labels, thereby constructing $\mathcal{D}_{U} = \{(I_{u}^t, Y_{u}^t)\}_{t=1}^{T}$.

In the second stage, the exemplar $\mathcal{D}_{E}$, the synthetic dataset $\mathcal{D}_{S}$ and the unlabeled dataset $\mathcal{D}_{U}$ are used as training data to train $\mathcal{N}_{e}$ by computing four losses as follows: $$L_{s2} = \lambda_{e}L_{e}+\lambda_{s}L_{s}+\lambda_{c}L_{c}+\lambda_{u}L_{u}
.
\label{stage3loss}$$ As a result, this stage exploits unlabeled data with their pseudo-labels.

In Eq. \[stage2loss\] and Eq. \[stage3loss\], we define $L_{e}=\mathcal{L}_{seg}(\hat{Y_{e}},Y_{e})$, $L_{s}=\mathcal{L}_{seg}(\hat{Y_{s}},Y_{s})$ and $L_{u}=\mathcal{L}_{seg}(\hat{Y_{u}},Y_{u})$, where $\hat{Y}_{e,s,u}$ indicates the output of the segmentation networks in the training stage. $\mathcal{L}_{seg}$ is defined as follows: $$\mathcal{L}_{seg}(\hat{Y},Y)=0.5*l_{ce}(\hat{Y},Y)+0.5*l_{dice}(\hat{Y},Y)
,
\label{eqseg}$$ where $l_{ce}$ is the cross-entropy loss function and $l_{dice}$ is the Dice loss function. After two-stage training, $\mathcal{N}_{e}$ can be directly used for inference.

Experimental Results
====================

Implementation Details
----------------------

We adopt a U-Shape transformer-based structure @wang2021mixed as the basic structure of $\mathcal{N}_{s}$ and $\mathcal{N}_{e}$ in the proposed ELSNet. The weights of the proposed ELSNet are randomly initialized. During all training stages, the input size is set to 224$\times$224 with random rotation and flipping. Adam optimizes the proposed ELSNet with a learning rate of 1e-4, a weight decay of 0.0001 and a learning rate of 1e-4. The batch size is set to 12, and the patch size is set to 16. We divide all 3D volumes into individual images during the testing stage for inference @wang2021mixed [@chen2021transunet]. We randomly select a sample containing all categories in the training set as the exemplar and conduct five cross-validations for each experimental configuration.

#### **Datasets and Evaluation metrics.**

We evaluate the proposed framework on two challenging medical image datasets: the Synapse multi-organ CT dataset and the Automated cardiac diagnosis challenge dataset. Synapse is a multi-label organ dataset containing 30 abdominal clinical CT cases. Each image is annotated with eight abdominal organs. We use 18 cases for training and 12 cases for testing @wang2021mixed [@chen2021transunet]. ACDC is a cardiac MRI dataset that contains 100 examples from MRI scanners. Each image is labeled into three categories: left ventricle, right ventricle, and myocardium. We used 70 samples for training, 20 for evaluation, and 10 for testing. We evaluated the proposed framework based on two metrics, namely, the Dice Similarity Coefficient (DSC) and the 95% Hausdorff Distance (HD95), which are widely used in the problem of semantic segmentation of medical images.

Quantitative Evaluation
-----------------------

**4.2.1 Ablation Studies**

  ----------------------- --------------- ------------- ------------------ -------------- --------------- ------------- ------------------ --------------
   \*<span>Method</span>                                                                                                                   
                           DSC$\uparrow$   $\Delta$DSC   HD95$\downarrow$   $\Delta$HD95   DSC$\uparrow$   $\Delta$DSC   HD95$\downarrow$   $\Delta$HD95
            BS                 0.112            -             154.60             -             0.142            -             74.20              -
           BS+BG               0.143                          110.35                           0.153                          62.18        
           +ESM                0.234                          120.59                           0.273                          38.35        
       +ESM+PCEM\_S1           0.264                        **101.00**                         0.355                          40.80        
      +ESM+PCEM\_S1&2        **0.315**                        109.70                         **0.410**                      **26.64**      
  ----------------------- --------------- ------------- ------------------ -------------- --------------- ------------- ------------------ --------------

  : \[tab:ablation\_modules\] Ablation study of the proposed modules. BS denotes the results of the baseline. BS+BG denotes the results obtained by adding background images based on the baseline. +ESM denotes the results obtained when the proposed ESM is used. +ESM+PCEM\_S1 denotes the results obtained when the two modules are used in stage 1. +ESM+PCEM\_S1&2 denotes the results obtained when the two modules are used in stage 1 and 2.

#### **Impact of the proposed modules.**

We first summarize the contribution of the proposed modules on two datasets in Table \[tab:ablation\_modules\]. Following the general experimental setup in @reiss2021every [@seibold2021reference], BS is the baseline that uses only the exemplar as supervision, which achieves 0.112 and 0.142. Adding background images as negative samples (BS+BG) brings minor improvements of 0.031 and 0.011 compared to these results. This demonstrates that background images are useful, but current solutions do not take advantage of them effectively. The performance obtained by the proposed exemplar-guided synthesis module "+ESM" is much better, with DSC values of 0.234 and 0.273. This phenomenon highlights the value of our synthetic dataset, which enriches the diversity of the sample and increases the generalization capability of the model. By including the proposed pixel-prototype based contrastive embedding module in stage 1, the results improve even more, reaching 0.264 and 0.355. Finally, the "+ESM+PCEM\_S1&2" method produces the best results on both datasets when all of the proposed modules are used in stage 1 and 2. This performance increase is mainly due to the framework's ability to simulate diverse appearance variants and explore discriminative information.

\[t\]

   $L_{e}$   $L_{s}$   $L_{c}^{e+s}$   $L_{c}^{e+s+u}$   $L_{u}$      RV          Myo         LV       DSC.Avg$\uparrow$   HD95.Avg$\downarrow$
  --------- --------- --------------- ----------------- --------- ----------- ----------- ----------- ------------------- ----------------------
   $\surd$      -            -                -             -        0.119       0.126       0.182           0.142                74.20
   $\surd$   $\surd$         -                -             -        0.192       0.265       0.362           0.273                38.35
   $\surd$   $\surd$      $\surd$             -             -        0.249       0.315       0.503           0.355                40.80
   $\surd$      -         $\surd$          $\surd$       $\surd$     0.263       0.323       0.488           0.359                32.49
   $\surd$   $\surd$         -             $\surd$       $\surd$     0.218       0.355       0.462           0.345                30.42
   $\surd$   $\surd$      $\surd$             -          $\surd$     0.290       0.354       0.529           0.390                30.29
   $\surd$   $\surd$      $\surd$          $\surd$       $\surd$   **0.293**   **0.374**   **0.563**       **0.410**            **26.64**

#### **Impact of different loss functions.**

To demonstrate the effectiveness of the loss functions, we summarize the results of various studies in Table \[tab:ablation\_loss\]. If $L_c$ is not used, we do not generate pseudo-labels for the unlabeled samples and only use exemplars and the synthetic dataset for training. The results in the second row show that creating the synthetic dataset can significantly improve the experimental results to 0.273. Then, using two modules simultaneously to train the model in stage 1 can substantially improve the results, as shown in the third row. Note that ignoring the synthetic dataset but still using the pseudo-labels of unlabeled data in stage 2 can obtain a value of 0.359. The experimental results reach 0.345 when the contrastive loss is only utilized in stage 2 and not in stage 1. In addition, the results obtained by not measuring the prototype contrastive loss in stage 2 are shown in the sixth row, with a value of 0.39. When all of the loss functions are utilized in stages 1 and 2, the best result is obtained, as shown in the last row, with a DSC value of 0.41 and an HD95 value of 26.64.

\[t\]

#### **Impact of different transformation strategies.**

To verify the effectiveness of several transformation strategies, we summarize the results of different variants in Table \[tab:ablation\_transformation\]. We use the same number of synthetic samples to ensure fair comparisons. Geometric (scaling, rotation) and intensity (blur, intensity variations) transformations are applied in this section. When none of the transformations is utilized, we paste the exemplar into different background images to build the synthetic dataset, making a minor improvement, as shown in the first row. When both transformations are applied to the exemplars, the experimental results are considerably improved to 0.311. This demonstrates that the strategy proposed in this paper for generating synthetic data is reasonable. Then, we use one of the transformations on both the exemplar and the background image, which are still effective and provide 0.315 and 0.339 DSC values, respectively. When neither Geo nor Int is applied to the background images, the experimental results still reach 0.370 and 0.346, as indicated in the second and third last rows. This indicates that the intensity diversity of the background is more important than geometry in our experiments. The best results are obtained by applying both transformations to the example and the background image simultaneously, as illustrated in the last row.

\[t\]

           Stage           Number   DSC.Avg$\uparrow$   HD95.Avg$\downarrow$      RV          Myo         LV
  ----------------------- -------- ------------------- ---------------------- ----------- ----------- -----------
   \*<span>Stage </span>     5            0.199                32.07             0.101       0.177       0.319
                             10           0.300                34.38             0.186       0.279       0.435
                             15           0.355                40.80             0.249       0.315       0.503
                             20           0.323                29.55             0.207       0.303       0.460
                             25           0.285                26.95             0.148       0.288       0.421
   \*<span>Stage </span>     5            0.211                55.17             0.153       0.185       0.296
                             10           0.320                22.02             0.138       0.333       0.488
                             15         **0.410**              26.64           **0.293**   **0.374**   **0.563**
                             20           0.377              **20.12**           0.279       0.360       0.491
                             25           0.364                23.58             0.247       0.362       0.482

#### **Impact of different numbers of synthetic samples for the same background.**

We evaluate the performance of the number of synthetic samples for the same background in Table \[tab:ablation\_synthetic\_samples\]. Different numbers of synthetic samples are generated by ESM, which are subsequently used in different training stages to obtain the results. Clearly, an intermediate number, <span>i.e. </span>, $15$, is beneficial for segmentation compared to $5,10,20,25$. It is reasonable that, along with the increasing number of synthetic samples, the proposed framework is prone to overfitting around the space of the exemplar sample distribution. In this case, a certain amount of synthetic samples can offer enough guidance in our framework. Therefore, this setting can ensure that the number of synthetic samples considers sufficiency and quality.

\[t\] ![image](Fig4.pdf){width="1\linewidth" height="30mm"}\

#### **Impact of key parameters.**

We summarize the results by using the proposed framework with different values of the parameters of Eq. \[stage3loss\] on the ACDC and Synapse datasets in Fig. \[fig:key\_parameters\]. We fix others to obtain the result when adjusting a certain $\lambda$. As shown in Fig. \[fig:key\_parameters\] (a), the proposed framework shows better performance on both datasets when the $\lambda_e$ is set to smaller values, <span>e.g. </span>0.2 and 0.4, than when they are set to larger values. This is reasonable, as a large value of $\lambda_e$ could easily overfit the network to the exemplar. In addition, when the value of $\lambda_s$ is set within a reasonable range, the experimental results can increase to some degree and reach their highest value at a setting of 1.2 (0.31 and 0.41), as shown in (b). Then, $\lambda_c$ peaks on both datasets when set to 5 and 10. Finally, better results are achieved when $\lambda_u$ is set to smaller values, while increasing it leads to a decrease in the overall trend. We analyse that this is mainly due to the degree of noise in the pseudo-labels.\

\[t\]

          Method           DSC.Avg$\uparrow$      RV          Myo         LV       HD95.Avg$\downarrow$      RV          Myo         LV
  ----------------------- ------------------- ----------- ----------- ----------- ---------------------- ----------- ----------- -----------
   MT-UNet@wang2021mixed         0.142           0.119       0.126       0.182            74.20             83.91       61.48       77.22
    MLDS@reiss2021every          0.189           0.144       0.165       0.258            50.03             72.13       30.20       47.77
          ELSNet               **0.410**       **0.293**   **0.374**   **0.563**        **26.64**         **47.63**   **16.58**   **15.73**

<span>**4.2.2 Comparison with the State-of-the-Art Methods**</span>\
We compared the proposed ELSNet with a few state-of-the-art image segmentation methods under the same experimental setting we adopted on the ACDC and Synapse datasets. The results are reported in Table \[tab:ACDC\] and Table \[tab:Synapse\], respectively. We can see the ELSNet outperforms the current state-of-the-art methods and produces the lowest HD95 and the highest DSC results on both datasets. On the ACDC dataset, the ELSNet achieves considerable improvements, thereby outperforming the second-best model (<span>i.e. </span>, MLDS) by 0.221 and 23.39 in terms of DSC and HD95, respectively. The MLDS @reiss2021every represents a medical image segmentation model that integrates deep supervision of decoded signals at different layers in semi-supervised scenarios. The superior performance of our ELSNet over this semi-supervised learning method validates the efficacy of our proposed exemplar-guided data synthesis and pixel-prototype based contrastive embedding learning.

        **Method**        DSC$\uparrow$   HD95$\downarrow$      Aor         Gal       Kid(L)      Kid(R)        Liv         Pan         Spl         Sto
  ---------------------- --------------- ------------------ ----------- ----------- ----------- ----------- ----------- ----------- ----------- -----------
   MTUNet@wang2021mixed       0.112            154.60          0.066       0.108       0.155       0.053       0.352       0.008       0.046       0.102
   MLDS@reiss2021every        0.221            159.26          0.057       0.147       0.306       0.183       0.638       0.038     **0.306**     0.090
          ELSNet            **0.315**        **109.70**      **0.319**   **0.372**   **0.381**   **0.219**   **0.784**   **0.067**     0.276     **0.104**

  : \[tab:Synapse\] Comparison results on the Synapse Dataset. The class average DSC and HD95 results and the DSC results for all individual classes are reported.

-.1in

For the Synapse benchmarks, the ELSNet considerably outperforms the MT-UNet @wang2021mixed and the MLDS @reiss2021every methods by reaching a DSC of 0.311 and an HD95 of 109.70. In this dataset, many foregrounds share a similar appearance with the background, and contain more organ categories with different sizes. However, the ELSNet still achieves the best results in most categories, especially in the Aor, Gal and Liv categories. Notably, the Aorta and Gallbladder organs are smaller and usually located in the middle of multiple organs, but the improvement over other methods is nearly 0.25 and 0.23, respectively. The performance improvements again demonstrate the efficacy of the two new modules in the proposed ELSNet.

Qualitative Evaluation
----------------------

Several visualized examples are presented in Fig. \[fig:vis\] to demonstrate the effect of the proposed framework on the medical image segmentation task. Typically, the tiny size of some organs makes segmentation challenging, not to mention that there is only one annotation. Furthermore, some images are vulnerable to background confusion, which exacerbates the difficulty of segmentation and leads to over-segmentation. Our framework can identify target organs from images using the proposed two modules to achieve segmentation with only one annotation. Compared with existing state-of-the-art methods, the proposed two modules help the ELSNet segment accurately, even under conditions of deformation, edge ambiguity, shape complexity and background clutter.

![image](Fig5.pdf){width="0.75\linewidth" height="98mm"}\

Conclusions
===========

This paper introduces a new experimental scenario, Exemplar Learning, and proposes a novel framework ELSNet that can perform segmentation with only one annotated image. The proposed exemplar-guided synthesis module shows that it can enrich and diversify training data by synthesizing samples from the given exemplar. Moreover, the proposed pixel-prototype based contrastive embedding module provides an effective way to increase the discriminatory ability of the model by contrastive self-supervised learning. A two-stage training process shows that it can explore unlabeled data via pseudo-labels. Experimental results demonstrate that the proposed framework is effective and outperforms existing methods. We conclude that it is valuable to raise a new scenario and perspective in tackling the medical image segmentation problem, inspiring future problems of limited annotation.

{'source': 'arXiv_out/2204'}