---
abstract: 'The consistency loss has the potential for semi-supervised learning in segmentation and localization with various recent problems as well as classification. The conventional methods based on consistency loss, however, are inefficiently applied to semantic segmentation task regarded as pixel-wise classification; the consistency loss is calculated only in matched pixel pairs and averaged out. To this end, we propose a *structured consistency loss* for semi-supervised semantic segmentation to promote the consistency by considering the inter-pixel correlation. Specifically, the proposed method dramatically improves the computational complexity in the collaboration with cutmix [@yun2019cutmix]. It is shown via experiments that the Cityscapes benchmark results with validation data and test data are 81.9mIoU and 83.84mIoU respectively. This ranks the *first* place on the pixel-level semantic labeling task of Cityscapes benchmark suite. To the best of our knowledge, this is the first study to achieve the state-of-the-art performance with semi-supervised learning approach in semantic segmentation which provides the insight of the applicability of the semi-supervised technique to the various types of learning-based systems.'
bibliography:
- 'bib.bib'
title: 'Structured Consistency Loss for Semi-Supervised Semantic Segmentation'
---

Introduction
============

With the deep learning approach of @krizhevsky2012imagenet introduced in ILSVRC-2012 [@ILSVRC15], deep learning-based image processing has acquired tremendous progress. Especially, the supervised learning has been paid increasing attention since the learning performance of the network is greatly affected by both quality and quantity of the labeled data. Meanwhile, the supervised learning requires the manually labeled data, which is time-consuming and expensive. When it comes to semantic segmentation, pixel-by-pixel annotation is required, which particularly leads to the exorbitant preparation of the labeled data. Recently, a various techniques such as active learning [@mackowiak2018cereals], interactive segmentation [@maninis2018deep], weakly-supervised leaning [@lee2019ficklenet] and so on have been developed to solve the labeling cost problem in semantic segmentation.

Semi-supervised learning has been introduced to address the ever-increasing size of modern data combined with the difficulty of obtaining label. In particular, semi-supervised learning improves the network performance with the use of relatively smaller number of labeled data, when a large number of unlabeled data is available. A simple build up of unlabeled data can be easily collected from various data sources via web crawling, vehicle logging, and etc. Especially, the manual labelling process to allow to make the collected unlabeled data useful in learning process is costly compared to other tasks in the progress of semantic segmentation. To address the cost issue of labeling, the semi-supervised learning technique best suits the semantic segmentation which requires expensive labeled data.

Prior studies [@tarvainen2017mean; @xie2019unsupervised] employ the semi-supervised learning techniques for image classification task in a various ways, and show the significant improvement in accuracy. However, the semantic segmentation is a different task compared to the classification in that while the prediction of classification results in a class vector, the semantic segmentation performs the structured prediction per regional location and predicts structural characteristics of regions. Prior studies [@liu2019structured; @xie2018improving] investigate the between-pixel relationship and verify that the structural relationship between pixels is important in semantic segmentation.

In this paper, we propose a semi-supervised learning with the structured consistency loss for semantic segmentation. The proposed loss forces the predictions of segmentation network consistent in not only pixel-wise, but also inter-pixel relationship, which allows the network to learn more powerful generalization capabilities to predict in harmony with neighboring pixels. Furthermore, when cutmix [@yun2019cutmix] is incoporated, network can get the better generalization performance with lowering GPU memory utilization by restricting the regions to calculate the structured consistency loss in cutmix box. Via numerical results, it is shown that the proposed method ranks the first place with mIOU 83.84 in the Cityscapes benchmark pixel-level semantic labeling task [@Cordts2016Cityscapes]. We note that this is the first study to verify that the semi-supervised learning can achieve the state-of-the-art performance in semantic segmentation using the Cityscapes benchmark. In addition, the proposed semi-supervised learning technique can be applied in parallel with other researches for further improvements since our contribution is not in network architecture, but in learning techniques.

Related Work {#related_works}
============

**Semantic Segmentation**.

The early period of semantic segmentation approaches are mostly based on Fully Convolutional Network (FCN) [@sermanet2013overfeat; @long2015fully]. To improve the performance of earlier segmentation models, the loss of spatial information is mediated with the use of encoder-decoder architecture [@noh2015learning; @ronneberger2015u; @badrinarayanan2017segnet] or dilated convolution (a.k.a. Atrous Convolution) expanding the receptive field [@chen2014semanticdeeplabv1; @yu2015multi]. For the further enhancement on the localization performance, [@chen2017deeplabv2] applies the Atrous Spatial Pyramid Pooling (ASPP) in semantic segmentation, and PSPNet [@zhao2017pyramid] proposes a feature pyramid pooling module to gather global contextual information around the image object or stuff. More recently, @chen2018encoderdeeplabv3+ suggestes the well-organized architecture with combining encoder-decoder architecture and dilated convolution, which have been followed by many subsequent methods to achieve state-of-the-art performance [@takikawa2019gated; @zhuang2018dense; @li2019global]. @zhu2019improving presents the inspiring improvement of results thanks to labelling of video image using temporal information, boundary relaxation loss to address the boundary issue and class uniform sampling for the class imbalance problem.\
**Semi-Supervised Learning**.

In recent years, the semi-supervised method has become a one of the most prominent theme, but is limitedly employed to classification task in earlier times. The usage of loss function computed on unlabeled data encourages the model to enhance generalization ability to unseen data in same domain. @grandvalet2005semi propose the entropy minimization loss to verify that the decision boundary tends to lie on a low density region of class distribution. The consistency loss is suggested in @laine2016temporal so as to encourage the model to produce the same output distribution when its inputs are perturbed. The consistency regularization loss plays a breakthrough role in the follow-up semi-supervised researches. To employ the consistency loss efficiently, exponential-moving-average (EMA) technique which builds the teacher network by accumulating weights of student network to generate the more accurate guessed label is invented by @tarvainen2017mean. MixMatch [@berthelot2019mixmatch] combines entropy minimization, consistency loss, and MixUp regularization [@zhang2017mixup] for the generalization performance improvement of the network. Additionally, the results of @xie2019unsupervised nearly match for the performance of models trained on the full sets of CIFAR-10 with only using 10 % of this dataset, thanks to sophisticated augmentation method with realistic noise.\
**Structured Prediction**.

The prediction results of semantic segmentation have an equal shape of input image, and also the prediction vector of each pixel has a strong correlation with each other, especially close one. In @xie2018improving, a local pair-wise (8-neighbors) distillation is used to make an efficient feature distillation. @liu2019structured presents an impressive progress of feature distillation by using pair-wise distillation, where the total inter-pixel similarities are calculated in a specific feature map, and the feature of teacher is distilled to student.

The knowledge distillation method which is similar to both the purpose and implementation of the consistency loss, is used to train the student network by forcing to resemble with the teacher network. Accordingly, the distance functions are usually utilized as a loss function by both the knowledge distillation and consistency loss. The consistency loss with cutmix [@yun2019cutmix] for semantic segmentation has been introduced in @french2019consistency. The authors in [@french2019consistency] investigate the difference between classification and semantic segmentation in terms of the low density region. The low density region lies in the boundary of class distribution in classification, while it lies in a regional boundary of object in semantic segmentation. They select the cutmix method instead of mixup to conserve the local boundarys of image, but with a conventional consistency loss.

Methodology
===========

Since semantic segmentation can be regarded as a pixel by pixel classification, it is reasonable to simply apply the conventional consistency loss which is useful for the classification to the semantic segmentation. The recent study [@french2019consistency] has made improvements by applying the consistency loss to the semantic segmentation. However, different from classification, the semantic segmentation has a characteristic of structured prediction that the predictions of pixels have correlation with each other. Therefore, if the existing consistency loss is used as it is, it is difficult to achieve the high performance improvement without consideration on the characteristic of semantic segmentation. In order to boost the performance of semi-supervised semantic segmentation, in this section we introduce the structured consistency loss, encouraging the network to predict properly by focusing on inter-pixel relationship in cutmix box.

Overall Training
----------------

Overall training of our network is composed of labeled, and unlabeled image in a batch simultaneously. The total loss $\mathcal{L}_{tot}$ is therefore a semi-supervised loss given as $$\mathcal{L}_{tot} = \mathcal{L}_{x} + \mathcal{L}_{u},
\label{eq1}$$

<span>2</span>

![The calculation of the consistency loss is described. The consistency loss is calculated in each paired pixel and averaged out. For the simplicity, Image size is reduced to 10 x 10 and has three cutmix boxes whose boundary is red line.<span data-label="Figure2"></span>](images/fig_B_0429.JPG){width="1.0\linewidth"}

where $\mathcal{L}_{u}$ is a loss with unlabeled image $u$ and $\mathcal{L}_{x}$ is a supervised cross entropy loss with boundary label relaxation [@zhu2019improving] for semantic segmentation with labeled image $x$ and written as $$\mathcal{L}_x = -log\sum\limits_{C\in \mathcal{N}_w}P(C).
\label{eq2}$$ $P(\cdot)$ and $\mathcal{N}_w$ are softmax probability for each class, and the set of classes within a $w$ by $w$ window boundary region, respectively. Note that $\mathcal{L}_x$ reduces to the standard cross entropy loss with one-hot label when $w$ equals one, i.e., $w$=1.

Architecture for unsupervised training
--------------------------------------

The cutmix augmentation is originally designed for classification task, where it mixes samples by cutting box and pasting from one sample into another. Following @french2019consistency, we use the cutmix augmentation method for semantic segmentation by generating $N$ boxes with random size and position. The total area of boxes is approximately the half of the image dimension to make balance between two images $u_a$ and $u_b$.

Our main architecture of unsupervised approach is described in Fig.\[Figure1\]. First, we generate randomly augmented input images ($\acute{u_a}$, $\acute{u_b}$) from the original images ($u_a$, $u_b$) by random augmentation used in @zhu2019improving except the cutmix, and create a cutmix image ($u_m$) from the augmented images. The teacher network makes prediction results ($p_t(\acute {u_a}) $, $p_t(\acute{u_b}) $) from the input images ($\acute{u_a}$, $\acute{u_b}$) and cutmix the results ($p_{t}(\acute{u_{a}},\acute{u_{b}})$). The student network uses the cutmix results ($p_{t}(\acute{u_{a}},\acute{u_{b}})$) as a *guessed label* introduced by @berthelot2019mixmatch. By getting the cutmix image ($u_m$) as input, the student network makes prediction ($p_s(u_m)$). Finally, we calculate two consistency losses using the prediction of student network and teacher network. Note that teacher and student network have the same network architecture, but the weights of teacher network are not from its own weights using gradient update, but from the student network by EMA manipulation. The guessed label can promote the generalization performance to the student network since the label is made by two original images information when the input to the student network is a complex image as several patches stuck on. The unlabeled loss term is separated into two parts such as consistency loss and structured consistency loss. The formula of unlabeled loss is given as follows: $$\mathcal{L}_{u} = \lambda_c\mathcal{L}_{c} + \lambda_{sc}\mathcal{L}_{sc},
\label{eq3}$$

$$\label{eq4}
\mathcal{L}_{c} = \frac{1}{H \times W} \sum\limits_{i \in \mathbb{T}} ||p_s(u_{m,i})-p_{t}(\acute{u_{a,i}},\acute{u_{b,i}})||^2,$$

where $\lambda_{c}$ and $\lambda_{sc}$ are hyperparameters for each loss term of $\mathcal{L}_{c}$ and $\mathcal{L}_{sc}$, respectively, and $\mathcal{L}_{sc}$ is the structured consistency loss which would be addressed in more detail in the following subsection. $\mathcal{L}_{c}$ is the conventional consistency loss as an average of pixel-wise squared L2 loss described in Fig.\[Figure2\] and Eq.\[eq4\]. $p_s(u_{m,i})$ represents a prediction of student network for the *i*th pixel of cutmix unlabeled image $u_m$, while $p_{t}(\acute{u_{a,i}},\acute{u_{b,i}})$ represents a cutmix of predictions of the teacher network for the *i*th pixel for unlabeled images $\acute{u_a}$ and $\acute{u_b}$. We denote a set of all the pixel indices in the image as $\mathbb{T}=\{1,2,...,H\times W\}$,

Structured consistency loss
---------------------------

As explained in section \[related\_works\], we utilize a concept of pair-wise knowledge distillation technique suggested by @liu2019structured for the structured consistency loss. Although the knowledge distillation and the consistency loss differs in underlying philosophy, the concept of consideration on inter-pixel relationship of pair-wise knowledge distillation is the great help in extending the conventional consistency loss into the structural one. The proposed structured consistency loss can be written as $$a_{ij} = \mathbf{p}_i^{T}\mathbf{p}_j/(||\mathbf{p}_i||||\mathbf{p}_j||),
\label{eq5}$$

$$\mathcal{L}_{sc} = \frac{1}{(H \times W)^2}\sum\limits_{i\in\mathbb{T}}\sum\limits_{j\in\mathbb{T}}||a_{ij}^s - a_{ij}^t||^2 ,
\label{eq6}\\$$

where $a_{ij}^{t}$ and $a_{ij}^{s}$ denote the similarity between the *i*th pixel and the *j*th pixel produced from the teacher network and student network, respectively, and $\mathbf{p}_i$ represents the prediction vector of the *i*th pixel. However, the structured consistency loss derived from Eq.\[eq6\] is not efficient because the most of pixel pairs are far from each other, have very low correlation, and therefore have little effect on performance. Moreover, Eq.\[eq6\] requires extremely high computation cost for computing all of pixel pairs in prediction map.

![The calculation of structured consistency loss is described. Assume that the above boxes are derived from teacher network, while the below boxes are derived from student network. Inter-pixel cosine similarity is calculated at each cutmix box. The hatched area of left cutmix box which is covered by the other box is not included in calculation. For simplicity, we set $N_{box}$=2.](images/fig_C_0429.JPG){width="1.0\linewidth"}

\[Figure3\]

In order to handle the above problems, we employ the cutmix augmentation, which limits the pixel pairs inside the local region and reduces computing complexity dramatically. By calculating the inter-pixel cosine similarity within $N$ cutmix box separately, we can reduce the computing complexity approximately $N$ times. The detailed operation is described in Fig.\[Figure3\]. Moreover, by using cutmix, the network can learn the ability to make the accurate predictions with limited local information, which can provide the prediction similar to that of global area. The structured consistency loss with cutmix is represented as $$a_{n,ij} = \mathbf{p}_{n,i}^{T}\mathbf{p}_{n,j}/(||\mathbf{p}_{n,i}||||\mathbf{p}_{n,j}||),
\label{eq7}$$

$$\mathcal{L}_{sc} = \frac{1}{N}\sum\limits_{n=1}^{N}\frac{1}{|\mathbb{T}_n|^2}\sum\limits_{i\in\mathbb{T}_n}\sum\limits_{j\in\mathbb{T}_n}||a_{n,ij}^s - a_{n,ij}^t||^2,
\label{equ8}$$

where $a_{n,ij}$ represents the similarity between the *i*th pixel and the *j*th pixel in *n*th cutmix box, $\mathcal{L}_{sc}$ is the structured consistency loss which is the sum of $N$ cutmix boxes calculation, $N$ is the number of cutmix boxes, and $\mathbb{T}_n = \{1,2,3,...,H_n \times W_n\}$ denotes a set of all the pixel indices in the *n*th cutmix box except for the region covered by other cutmix box.

To make the structured consistency loss more efficient, we restrict the number of cutmix boxes to $N_{box}$ and the maximum number of pairs to be calculated in each box to $N_{pair}$. In Fig.\[Figure3\], we eliminate the region covered by another cutmix box, and also randomly drop out the similarity vectors excessing over $N_{pair}$. Then, we derive the efficient and effective structured consistency loss as follows:

$$\mathbb{T}_{n,drop}^{s} = Drop\boldsymbol{\langle}\{(i,j)|\forall i\in \mathbb{T}_n, \forall j\in \mathbb{T}_n \}, N_{pair}\boldsymbol{\rangle},
\label{equ9}$$

$$\mathcal{L}_{sc} = \frac{1}{N_{box}}\sum\limits_{n=N-N_{box}+1}^{N}\frac{1}{|\mathbb{T}_{n,drop}^{s}|}\sum\limits_{(i,j)\in\mathbb{T}_{n,drop}^{s}}||a_{n,ij}^s - a_{n,ij}^t||^2,
\label{equ10}$$

where the customized function $Drop\boldsymbol{\langle} \mathbb{X}, n \boldsymbol{\rangle}$ randomly removes the elements of $\mathbb{X}$ to keep the maximum number of elements as $n$. In the process of attaching the cutmix box, the box attached earlier is likely to be covered by the box attached behind, and then the structed consistency loss is calculated using posterior $ N_{box}  $ of boxes. In addition, the loss is not calculated via masking the area covered by another cutmix box. By doing this, only the pixels that are structurally-relevant within the cutmix box can learn the relationship each other.

Experiments
===========

Implementation details {#exp_imp}
----------------------

**Network Structures**.

We adopt a state-of-the-art segmentation architecture, Deeplabv3+ with WideResNet38 from @zhu2019improving. *Output stride* of the last layer of encoder and the low level feature transferred to decoder is 8, and 2 respectively. EMA [@tarvainen2017mean] is used for the teacher network to generate the guessed label. Inference with the test sets and the validation sets is also carried out by this model. The parameter value of EMA weight is set to 0.999 which is averaged out in every training steps by following [@tarvainen2017mean].\
**Training Procedure**.

We use a SGD optimizer with the polynomial learning rate policy. Following the setup of @zhu2019improving, the initial learning rate is 0.002, the power set to 1.0, weight decay of 0.001 and momentum of 0.9. We use the synchronized batch normalization with a batch size of 2 per GPU, one for labeled data and the other for unlabeled one, on 8 V100 GPUs. The training epoch is set to 175 based on the labeled data, while, for unlabeled data, we adopt cutmix augmentation. From the empirical experiments, we set the hyper-parameters $N$, $N_{box}$, $N_{pair}$ $\lambda_{c}$ and $\lambda_{sc}$ to 32, 16, 9000, 20 and 3, respectively. We use additional training skills such as Mapillary pre-training, class uniform sampling and boundary label relaxation [@zhu2019improving] so as to get the strong baseline network.\
**Cityscapes**.

Cityscapes is a widely accepted validation medium among recent studies [@zhu2019improving; @french2019consistency] owing to its high-quality labeled image database. Cityscapes offers the pixel-level annotations of 5,000 images and coarse annotations of 20K images. The reliable pixel-level annotations are also split into three subsets, that is, training, validation and test, each of which consists of 2,975, 500 and 1,525 images, respectively, and gauge the performance of our methodology. We also use coarsely annotated images for class uniform sampling to overcome the imbalance between classes, which are also applied with ignoring the annotations associated with unlabeled images in unlabeled training phase for greater learning exposure.

\[test\_cityscape\]

<span>l||c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c||c</span>

Method

&

road

&

s.walk

&

build.

&

wall

&

fence

&

pole

&

t-light

&

t-sign

&

veg

&

terrain

&

sky

&

person

&

rider

&

car

&

truck

&

bus

&

train

&

motor

&

bike

&

mIoU

\

PSPNet [@zhao2017pyramid]

&

98.7

&

86.9

&

93.5

&

58.4

&

63.7

&

67.7

&

76.1

&

80.5

&

93.6

&

72.2

&

95.3

&

86.8

&

71.9

&

96.2

&

77.7

&

91.5

&

83.6

&

70.8

&

77.5

&

81.2

\

DeeplabV3+ [@chen2018encoderdeeplabv3+]

&

98.7

&

87.0

&

93.9

&

59.4

&

63.7

&

71.4

&

78.2

&

82.2

&

94.0

&

73.0

&

95.8

&

88.0

&

73.0

&

96.4

&

78.0

&

90.9

&

83.9

&

73.8

&

78.9

&

82.1

\

Gated-SCNN [@takikawa2019gated]

&

98.7

&

87.4

&

94.2

&

61.9

&

64.6

&

72.9

&

**79.6**

&

82.5

&

**94.3**

&

73.3

&

**96.2**

&

88.3

&

74.2

&

**96.6**

&

77.2

&

90.2

&

87.7

&

72.6

&

79.4

&

82.8

\

DRN-CRL [@zhuang2018dense]

&

98.8

&

87.7

&

94.0

&

**65.0**

&

64.2

&

70.2

&

77.4

&

81.6

&

93.9

&

73.5

&

95.8

&

88.0

&

74.9

&

96.5

&

**80.8**

&

92.1

&

88.5

&

72.1

&

78.8

&

82.8

\

GALD-Net [@li2019global]

&

98.8

&

87.7

&

94.2

&

65.0

&

**66.7**

&

**73.1**

&

79.3

&

82.4

&

94.2

&

72.9

&

96.0

&

**88.4**

&

**76.2**

&

96.5

&

79.8

&

89.6

&

87.7

&

**74.1**

&

**79.9**

&

83.3

\

Video Propagation [@zhu2019improving]

&

98.8

&

87.8

&

94.2

&

64.0

&

65.0

&

72.4

&

79.0

&

82.8

&

94.2

&

74.0

&

96.1

&

88.2

&

75.4

&

96.5

&

78.8

&

**94.0**

&

**91.6**

&

73.7

&

79.0

&

83.5

\

Ours

&

**98.8**

&

**88.2**

&

**94.3**

&

64.5

&

65.3

&

72.6

&

79.3

&

**82.9**

&

94.2

&

**74.2**

&

96.1

&

88.4

&

75.6

&

96.6

&

79.5

&

93.7

&

91.2

&

73.7

&

79.4

&

**83.6**

\

Tencent AI Lab

&

98.6

&

86.9

&

94.1

&

63.5

&

63.0

&

70.7

&

77.7

&

80.2

&

94.0

&

73.1

&

95.9

&

87.8

&

74.5

&

96.3

&

82.8

&

94.3

&

90.4

&

74.0

&

77.5

&

82.9

\

GGCF

&

98.8

&

87.8

&

94.1

&

66.0

&

66.1

&

71.1

&

78.4

&

82.2

&

94.1

&

**74.5**

&

95.8

&

88.0

&

74.0

&

96.5

&

79.9

&

92.4

&

90.8

&

71.8

&

78.4

&

83.2

\

iFLYTEK-CV

&

98.7

&

87.1

&

94.1

&

64.4

&

65.4

&

71.2

&

77.9

&

82.2

&

94.0

&

73.5

&

96.0

&

88.3

&

75.7

&

96.5

&

**83.3**

&

**94.7**

&

**92.4**

&

74.3

&

79.0

&

83.6

\

openseg-group

&

98.8

&

88.3

&

94.3

&

**66.9**

&

**66.7**

&

**73.3**

&

**80.2**

&

**83.0**

&

94.2

&

74.1

&

96.0

&

88.5

&

75.8

&

96.5

&

78.5

&

91.8

&

90.2

&

73.4

&

79.3

&

83.7

\

Ours+

&

**98.9**

&

**88.4**

&

**94.3**

&

65.2

&

65.9

&

72.8

&

79.5

&

83.0

&

**94.3**

&

74.3

&

**96.1**

&

**88.6**

&

**75.9**

&

**96.6**

&

79.3

&

93.8

&

91.5

&

**74.8**

&

**79.7**

&

**83.8**

\

Results
-------

![Qualitative results of our proposed method on the Cityscapes test set, where the pictures depict the predicted segmentation masks. <span data-label="test_inference"></span>](test.PNG){width="1.0\linewidth"}

The experimental results from our semi-supervised learning technique with the structured consistency loss are summarized in Table \[test\_cityscape\]. The experiments are conducted on Cityscapes test images with multi-scale strategy (0.5, 1.0 and 2.0), horizontal-flip, and overlapping-tile methods following [@zhu2019improving]. The performance results achieved by our proposed method (marked as "Ours" and "Ours+") rank the first place in the Cityscapes benchmark, including the results of unpublished studies. It is also noticeable that the result with only training set exhibits the highest performance result compared with the performance results of published methods. Graphical results on the Cityscapes test set are presented in Fig.\[test\_inference\].

We believe that the our observed performance is attributed to (i) the effectiveness of structured consistency loss added to semi-supervised learning in semantic segmentation and (ii) the usage of coarsely labeled images in unlabeled training phase. Extant studies of semantic segmentation use the coarsely labeled data only in the early training stage, or sporadically use to aid the segmentation of image with rare classes. From our numerical results, the coarsely labeled data can be fully exploited not only by using the coarsely label in restricted way, but also by semi-supervised learning approach.

Ablation Study
--------------

**Effect of Semi-Supervised Losses**.

In this section, we conduct the additional experiments to demonstrate the superiority and effectiveness of two consistency losses. We first train the baseline network with only the supervised method whose result is summarized in the fifth row of Table \[ablation\_baseline\]. It is shown in Table \[ablation\_baseline\] that semi-supervised learning is beneficial to the semantic segmentation because the performance increases 0.37% by adding consistency loss to the baseline, and performance further increases with the addition of the structured consistency loss to 81.90%, which verifies the significant improvement by our proposed method. The qualitative results have been visually provided in Fig.\[val\_compare\] by allowing us to gauge the qualitative performance. Thanks to the inter-pixel consistency processed with the structured consistency loss, the proposed method exhibits the less errors in small and visually confusing region(s).\
**Exponential-Moving-Average (EMA) Application**.

The EMA is a well-known method for its great generalization ability [@tarvainen2017mean; @berthelot2019mixmatch], beneficial to apply for both building a teacher network and employing in validation (or test) phase. In recent studies in semi-supervised learning, however, not all the studies apply EMA to the weight of teacher network, or the inference network which implies that there is no rule of thumb. In order to examine the performance dependency on EMA, we conduct an experiment whose results are summarized in Table \[ablation\_ema\]. When the EMA weight is used in the teacher network, mIoU increases 0.135% on average, while the use of EMA weight in validation seems to have no significant effect on performance. Since the use of EMA weight in teacher network and validation yields the greatest mIoU, we proceed to use the EMA weight in both.

<span>2</span>

\[ablation\_baseline\]

<span> m<span>3cm</span>|c|c</span> Method & mIoU (%) & Gain (%)\
DeepLabV3+ [@chen2018encoderdeeplabv3+] & 79.6 &

-\
PSPNet [@zhao2017pyramid] & 79.7 &

-\
Gated-SCNN [@takikawa2019gated] & 80.8 &

-\
Video Propagation [@zhu2019improving] & 81.4 &

-\

Baseline & 81.17 &

0.0\
+ Consistency loss & 81.55 & +0.38\
+ Structured Consistency loss & 81.90 & +0.35\

  -------------- -------------- -------
                                
     Teacher       Validation   
   $\mathsf{X}$   $\mathsf{X}$   81.77
   $\mathsf{X}$   $\mathsf{O}$   81.74
   $\mathsf{O}$   $\mathsf{X}$   81.88
   $\mathsf{O}$   $\mathsf{O}$   81.90
  -------------- -------------- -------

  : Comparison of mIoU depending on whether exponential-moving-average (EMA) is applied to both/either teacher network and/or validation on Cityscapes image.<span data-label="ablation_ema"></span>

![Qualitative results on the validation data of Cityscapes. From the left to the right are original images, supervised-only, the addition of consistency loss, the addition of structured consistency loss, and ground truth. Red box is zoomed-region where is the visually confusing.<span data-label="val_compare"></span>](val_compare_woabcde.PNG){width="1.0\linewidth"}

Conclusions
===========

In this paper, we propose the structured consistency loss in semi-supervised learning for semantic segmentation. With the cutmix augmentation, the structured consistency loss fully exploits the relationship among the local regions, and enhances the network generalization, which can also be simultaneously applied to the general network. It is verified via numerical results that our proposed method achieves the state-of-the-art premier performance in the Cityscapes benchmark suite, not only ranking the first place only among publication results, but also among all results including unpublished results. Additionally, it is verified that the semi-supervised learning is highly effective to solve practical real world problems under data-insufficiency when it is accompanied with cutmix augmentation as well as the structured consistency loss.

{'source': 'arXiv_out/2001'}