---
abstract: 'In this work, we introduce a new concept, named source-free open compound domain adaptation (SF-OCDA), and study it in semantic segmentation. SF-OCDA is more challenging than the traditional domain adaptation but it is more practical. It jointly considers (1) the issues of data privacy and data storage and (2) the scenario of multiple target domains and unseen open domains. In SF-OCDA, only the source pre-trained model and the target data are available to learn the target model. The model is evaluated on the samples from the target and unseen open domains. To solve this problem, we present an effective framework by separating the training process into two stages: (1) pre-training a generalized source model and (2) adapting a target model with self-supervised learning. In our framework, we propose the Cross-Patch Style Swap (CPSS) to diversify samples with various patch styles in the feature-level, which can benefit the training of both stages. First, CPSS can significantly improve the generalization ability of the source model, providing more accurate pseudo-labels for the latter stage. Second, CPSS can reduce the influence of noisy pseudo-labels and also avoid the model overfitting to the target domain during self-supervised learning, consistently boosting the performance on the target and open domains. Experiments demonstrate that our method produces state-of-the-art results on the C-Driving dataset. Furthermore, our model also achieves the leading performance on CityScapes for domain generalization.'
author:
- |
    Yuyang Zhao$^{\textcolor{mypink}{1}}$[^1] Zhun Zhong$^{\textcolor{mypink}{2}*}$ Zhiming Luo$^{\textcolor{mypink}{3}}$ Gim Hee Lee$^{\textcolor{mypink}{1}}$ Nicu Sebe$^{\textcolor{mypink}{2}}$\
    <span>$^{\textcolor{mypink}{1}}$ National University of Singapore</span>\
bibliography:
- 'references.bib'
title: 'Source-Free Open Compound Domain Adaptation in Semantic Segmentation'
---

Introduction
============

Deep learning has now achieved a remarkable success in fully-supervised semantic segmentation @fcn [@zhao2017pyramid; @deeplab], which, however, is relied heavily on the expensive dense pixel-wise annotations. One solution to lighten the labeling cost is unsupervised domain adaptation (UDA), which aims to transfer the knowledge of labeled synthetic data to unlabeled real-world data. Despite the effectiveness of existing UDA methods @adaptseg [@zhang2021prototypical; @zheng2021rectifying], they mainly consider the context of a single target domain, resulting in limited applications in the real-world. Indeed, the target domain may be captured from multiple data distributions without a clear separation and the system will unavoidably face instances from unseen domains. To investigate a more realistic domain adaptation problem, in this paper, we consider the setting of open compound domain adaptation (OCDA) @liu2020open for semantic segmentation. In OCDA, the unlabeled target domain is a compound of multiple homogeneous domains without domain labels. The adapted model is applied to test samples from the compound target domain and an open domain, where the open domain is unseen during training.

Existing UDA @adaptseg [@zhang2021prototypical; @advent] and OCDA @liu2020open [@park2020discover; @gong2020cluster] methods commonly require the use of the labeled source data during the whole training process. However, the source data are not always available due to data privacy. In addition, the source data are generally very large,

<span>r</span><span>0.51</span>

<span>0.51</span><span>p<span>1.2cm</span>|m<span>0.55cm</span>&lt;m<span>0.55cm</span>&lt;m<span>0.85cm</span>&lt;m<span>0.75cm</span>&lt;m<span>0.55cm</span>&lt;</span> & & Source & Unlabeled & & Open\
  & Data & Model & Target & Targets & Targets\
DA @adaptseg & & & & &\
SF-DA @liu2021source & && & &\
DG @DRPC & & & & &\
OCDA @liu2020open & & & & &\
SF-OCDA & & & & &\

which require plenty of storage space (, GTA5 @gta5$\approx$57GB). This further limits the applications of existing methods, especially when transferring to a lightweight self-driving device. Nevertheless, we can choose to maintain the pre-trained source model instead of the source data, enabling us to obey the data privacy policy and use much less storage space (, DeepLab-VGG16 @deeplab [@vgg]$\approx$120MB). These facts motivate us to introduce a more challenging but practical setting for OCDA, called source-free OCDA (**SF-OCDA**), where only the source pre-trained model and the unlabeled target data are available during the training of the target model. In the literature, source-free domain adaptation (SF-DA) has recently been developed in image classification @liang2020we [@kundu2020universal] and semantic segmentation @liu2021source for the single target case. However, as shown in Fig. \[fig:intro\] and Tab. \[tab:settings\], compared with SF-DA, our SF-OCDA demands not only adapting to data from multiple target domains but also considering the generalization performance on unseen domains.

![Illustration of source-free open compound domain adaptation (SF-OCDA). In the training stage, the model is first trained on the (synthetic) labeled data and then adapted to the (real-world) unlabeled target data. The data are **not available** during the target adaptation. In the testing stage, the learned model is used to predict the semantic segmentation results for samples from the and domains.<span data-label="fig:intro"></span>](figures/intro.pdf){width="0.97\linewidth"}

In SF-OCDA, the source data and target data are invisible to each other. In such context, we cannot align the domain distributions as traditional DA methods @adaptseg [@zhang2021prototypical; @zheng2021rectifying]. Instead, this paper introduces an effective two-stage framework for SF-OCDA, which consists of (1) training a generalized source model and (2) adapting the target model with self-supervised learning. In the first stage, we aim to learn a robust model, which can generalize well to different target domains. To achieve this goal, we propose the **C**ross-**P**atch **S**tyle **S**wap (CPSS), which can effectively augment the samples with various image styles. Specifically, CPSS first extracts the styles of patches in feature maps and then randomly exchanges the styles among patches by the instance normalization and de-normalization. In this manner, CPSS can prevent the model from overfitting to the source domain and thus significantly improve the generalization ability of the model. In the second stage, we adapt the target model by self-supervised learning. Specifically, we optimize the target model with the guide of pseudo-labels generated from the pre-trained source model, which can implicitly align the source and target distributions under the constraint of label consistency. Moreover, CPSS is also applied to reduce the influence of noisy pseudo-labels and to avoid overfitting to the target domain, which can further boost the performance on the compound and open domains. Our contributions are summarized as follows:

-   We introduce a new setting for semantic segmentation, *i.e.*, source-free open compound domain adaptation (SF-OCDA), which is an important yet unstudied problem. In addition, we propose an effective framework for solving SF-OCDA, which focuses on learning a generalized model during the stages of source pre-training and target adaptation.

-   We propose the CPSS, which diversifies the samples in the feature-level, to improve the generalization ability of the model in both source and target training stages. CPSS is a lightweight module without learnable parameters, which can be readily injected into existing segmentation models.

-   The proposed framework learned with the source-free constraint significantly outperforms the state-of-the-art methods on the OCDA benchmark. Our approach also surpasses the advanced domain generalization approaches on CityScapes.

Related Work
============

**Transfer Learning in Semantic Segmentation**. To tackle the expensive cost of collecting and labeling real-world data, transfer learning has attracted a widespread attention in semantic segmentation. Commonly, transfer learning methods are developed along two directions: unsupervised domain adaptation (UDA) and domain generalization (DG). UDA aims at transferring the knowledge from a labeled source domain to an unlabeled target domain. Existing UDA approaches can be roughly divided into two categories, , aligning domain distributions through adversarial learning @adaptseg [@advent; @luo2019taking; @du2019ssf] and self-training on the target domain @cbst [@zhang2021prototypical; @ma2021coarse; @zheng2021rectifying]. DG focuses on training a robust model with synthetic data, which can generalize well on unseen real-world target data. To reduce the large gap between synthetic data and the real-world data, DG methods usually augment the synthetic samples @DRPC [@FSDR] with the styles of ImageNet @imagenet or conditionally align the outputs @ASG [@CSG] between the segmentation model and the ImageNet pre-trained model. On the other hand, some works are proposed to learn domain-invariant features by removing domain-specific information @robustnet [@ibn] or feature augmentation @crossnorm. Recently, Liu  @liu2020open propose the setting of open compound domain adaptation (OCDA), which can be regarded as an extension of UDA and DG. In OCDA, the model trained with source and target data is used to evaluate samples from the compound target domain and unseen open domain. Liu  @liu2020open introduce a memory-based curriculum learning framework to improve the generalization on the compound and open domains. Park  @park2020discover and Gong  @gong2020cluster discover the latent target domains and align the source and latent domains with multiple domain discriminators. Different from these OCDA methods, this work investigates the OCDA under the source-free constraint and aims to learn a robust model by augmenting features with patch styles.

**Source-Free Domain Adaptation**. Hypothesis transfer learning (HTL) @kuzborskij2013stability aims to retain the prior knowledge in a form of hypothesis instead of data for the source domain. However, the main drawback of HTL is that it requires a small set of labeled target data. Inspired by HTL, source-free domain adaptation (SF-DA) @chidlovskii2016domain [@liang2019distant; @liu2021source; @liang2020source; @liang2020we; @kundu2020universal; @tian2021vdm; @hou2021visualizing] has recently flourished in the domain adaptation community. In SF-DA, instead of the source data, the source pre-trained model is provided in the target training stage. SHOT @liang2020we maintains the source hypothesis by fixing the trained classifier and maximizes mutual information of target outputs for distribution alignment. Kundu  @kundu2020universal generate negative samples by image composition, which are used to narrow domain shift and category gap during source training. In addition, an instance-level weighting mechanism is proposed for effective target adaptation. Lately, Liu  @liu2021source introduce source-free domain adaptation for semantic segmentation and utilize the batch normalization statistics of the source model to recover source-like samples. In this work, we introduce the source-free open compound domain adaptation (SF-OCDA) for semantic segmentation, extending SF-DA to a more realistic setting. The comparisons between SF-OCDA and existing adaptation settings are reported in Tab. \[tab:settings\].

Methodology
===========

![The framework of the proposed method. (1) The model is first trained on the labeled source domain. (2) We generate pseudo-labels by the source pre-trained model and train the target model in a self-training manner. In the second stage, we have no access to the source data. To improve the generalization ability of the model, we equip the model with the Cross-Patch Style Swap module in the two training stages, which augments features by exchanging styles among patches.<span data-label="fig:framework"></span>](figures/framework.pdf){width="0.95\linewidth"}

Preliminaries
-------------

In open compound domain adaptation (OCDA) @liu2020open, we are given a labeled (synthetic) source domain $\mathcal{S}$ and an unlabeled (real) compound target domain $\mathcal{T}$. The goal is to train a model that can accurately predict semantic labels for instances from the compound and open target domains. Specifically, $\mathcal{S}$ includes $N_S$ images $x^s_i\in \mathbb{R}^{H^s\times W^s \times 3}$ and their corresponding semantic labels $y^s_i\in \mathbb{R}^{H^s\times W^s}$ of $C$ classes. $\mathcal{T}$ contains $N_T$ images $x^t_i\in \mathbb{R}^{H^t\times W^t \times 3}$ of multiple homogeneous domains without semantic and domain labels. In this paper, **we consider the setting of source-free OCDA (SF-OCDA)**, which imposes an extra constraint that only the pre-trained source model, instead of the source data, is available for training the target model together with the unlabeled target data.

Overview
--------

In this Section, we propose an effective framework for SF-OCDA, which separates the training process into two stages: (1) training a generalized source model and (2) adapting a target model with self-supervised learning. We also introduce the Cross-Patch Style Swap (CPSS) to augment features with various patch styles, which can significantly improve the generalization ability of the model in both training stages. The pipeline of our framework is shown in Fig. \[fig:framework\]. Next, we first introduce our CPSS module (Sec. \[sec:cpss\]) and then present the proposed training strategy (Sec. \[sec:training\]) in detail.

\[t\]<span>0.42</span> ![Visualization of style distributions for (a) source domain and (b) target domain. We use the concatenation of mean and standard deviation of the feature map after the first block of VGG-16 @vgg as the style feature and show the 2D embeddings by t-SNE @tsne. Zoom in for details.<span data-label="fig:style_tsne"></span>](figures/src_tsne.png "fig:"){width="100.00000%"}

\[t\]<span>0.42</span> ![Visualization of style distributions for (a) source domain and (b) target domain. We use the concatenation of mean and standard deviation of the feature map after the first block of VGG-16 @vgg as the style feature and show the 2D embeddings by t-SNE @tsne. Zoom in for details.<span data-label="fig:style_tsne"></span>](figures/tgt_tsne.png "fig:"){width="100.00000%"}

Cross-Patch Style Swap {#sec:cpss}
----------------------

**Motivation**. Image style variation is an important factor that influences the model performance in semantic segmentation. Although the synthetic data are built to simulate the real-world images, the styles of synthetic images are still very different from that of the real ones. Therefore, the model trained on the synthetic data will be sensitive to the real style variations and thus produces poor performance on real images. To this end, we attempt to learn a robust model, which is insensitive to style variations, by augmenting the training samples with diverse styles.

In order to implement style augmentation, the key is extracting style factors from images. To achieve this goal, we draw inspiration from the style transfer @in [@dumoulin2016learned], which obtains image styles by extracting the mean $\mu$ and standard deviation $\sigma$ of the feature map in a network. In Fig. \[fig:style\_tsne\], we visually verify the feasibility of using the $\mu$ and $\sigma$ as the style features in semantic segmentation. It is clear that images of various styles  (, day and night) can be well-separated by the style features. In addition, AdaIN @adain shows that an input sample can be transferred to an arbitrary style while remaining the semantic content, by replacing the style features, formulated as: $$\text{AdaIN}(x, y)=\sigma(y)\left(\frac{x-\mu(x)}{\sigma(x)}\right)+\mu(y),$$ where $\mu(.)$ and $\sigma(.)$ denote the mean and standard deviation of the input feature map, respectively. $x$ and $y$ are two feature maps that provide the semantic content and the image style, respectively. Inspired by AdaIN, we propose two style augmentation operations based on the style features of image patches for training a robust segmentation model.

**Intra-Image Cross-Patch Style Swap.** In the self-driving scenario, different patches (*e.g.*, up and down) of a frame may include different objects, such as sky, vehicle, road and fence, making these patches present different styles. Intuitively, we can generate a new stylized sample by exchanging the style features of different patches. Hence, we propose the intra-image Cross-Patch Style Swap. Specifically, the feature map of an image is first separated into $n=n_h\times n_w$ patches: $$F =
{\left[                
  \begin{array}{ccc}   
    F_{1,1} & \cdots   & F_{1,n_w}   \\
    \vdots & \ddots   & \vdots  \\
    F_{n_h,1}  & \cdots & F_{n_h,n_w}  \\
  \end{array}
\right]}.$$

After that, each patch is normalized by the mean and standard deviation of itself, and de-normalized by the feature style of a random patch, formulated by: $$\label{eq:intra-cpss}
F^{'}_{i,j}=\sigma(\tilde{F}_{i,j})\left(\frac{F_{i,j}-\mu(F_{i,j})}{\sigma(F_{i,j})}\right)+\mu(\tilde{F}_{i,j}),$$ where $F^{'}_{i,j}$ denotes the style swapped counterpart of $F_{i,j}$. $\tilde{F}_{i,j}$ denotes the shuffled patch that provides the style feature.

**Inter-Image Cross-Patch Style Swap.** Although the intra-image CPSS can enrich the styles of a feature map, the model can easily remember the intra-image style variations after several training epochs, which will limit the effectiveness of the CPSS. However, the patch styles vary greatly among different images, which can be used to further enhance the style diversity during CPSS. Taking this into consideration, we introduce the inter-image CPSS, which collects style features from all the patches in a mini-batch with $B$ samples and exchanges these styles ($B\times n$) among all patches. We re-formulate Eq \[eq:intra-cpss\] as: $$F^{'}_{k,i,j}=\sigma(\tilde{F}_{k,i,j})\left(\frac{F_{k,i,j}-\mu(F_{k,i,j})}{\sigma(F_{k,i,j})}\right)+\mu(\tilde{F}_{k,i,j}),$$ where $F^{'}_{k, i,j}$ denotes the swapped counterpart of patch $F_{i,j}$ in the $k$th sample. $\tilde{F}_{k,i,j}$ denotes a randomly selected patch that provides the style feature.

CPSS is injected into several layers of the backbone, which is activated in the training stage with a probability of $\beta$ and is not used in the testing stage.

**Photometric Transformation**. In practice, the brightness, contrast and saturation of the frame vary in different situations. For example, images are brighter in sunny morning while the contrast is stronger in snowy weather. In addition, there may exist blurry images caused by the rainy weather. Consequently, we randomly apply photometric transformation to the input images, including color jitter, Gaussian blur and grayscale, to simulate the real-world style various, which can further improve the effect of CSPP.

Model Training {#sec:training}
--------------

As shown in Fig. \[fig:framework\], our framework includes two stages, , the source training stage and the target training stage, where the source data and target data are used independently in their own stages.

**Stage-I: Source Training.** In this stage, we aim at training a generalized model with synthetic labeled source domain $\mathcal{S}$. We adopt the cross-entropy loss to train the model, formulated as: $$L_{seg} = - \sum\limits_{m=1}^{H \times W} \sum\limits_{c=1}^C y^s_{m,c} \log p^s_{m,c},$$ where $y^s_{m,c}$ denotes the ground truth for the $m$th pixel and $p^s_{m,c}$ denotes the softmax probability of this pixel belonging to the $c$th class. Importantly, we employ the proposed CPSS along with photometric transformation to augment the samples in both feature- and image-levels, which can effectively improve the generalization ability of the source model.

**Stage-II: Target Training.** For SF-OCDA, source data are not available in this stage. Instead, we are given the source pre-trained model and the unlabeled compound target domain $\mathcal{T}$ to learn a target model that can perform well on both compound and open domains. In this stage, the target model is cloned from the source pre-trained model and trained in a self-supervised manner.

Specifically, we first generate pseudo-labels based on the predictions of the source pre-trained model by maximum probability threshold (MPT) @li2019bidirectional. MPT estimates class thresholds based on the top $q$% pixels of each class and a predefined threshold $\tau$. The pseudo-labels are then assigned to pixels where the prediction values of the dominant classes are higher than the corresponding class thresholds.

With the pseudo-labels, we employ the cross-entropy loss to enforce the consistency between the source and target outputs: $$L_{ssl} = - \sum\limits_{m=1}^{HW} \sum\limits_{c=1}^C \hat{y}^t_{m,c} \log p^t_{m,c},$$ where $p^t_{m,c}$ is the prediction of the target model and $\hat{y}^t_{m,c}$ is the generated pseudo-label. Note that, we only update the model with pixels that are assigned with pseudo-labels, and ignore the others.

Similar to Stage-I, we also adopt CPSS and the photometric transformation to train the target model, which brings two advantages. First, the negative impact of noisy pseudo-labels can be reduced by training on samples with more augmentations @sohn2020fixmatch. Second, CPSS can prevent the model from overfitting to the styles of the target domain, leading the model to be more robust to style variations. These two advantages improve the model performance on the target compound and open domains.

Experiments
===========

Experimental Setup {#sec:details}
------------------

**Datasets.** Following @liu2020open, we use the synthetic image data GTA5 @gta5 as the source domain, the rainy, snowy, and cloudy images in C-Driving @bdd100k [@liu2020open] as the compound target domain, and the overcast images in C-Driving as the open domain. To further measure the generalization ability of models, we additionally use Cityscapes @cityscapes as an extended open domain. GTA5 includes 24,966 training images with a resolution of 1914$\times$1052. C-Driving consists of 14,697 unlabeled training images and 1,430 testing images, where the image size is 1280$\times$720. Cityscapes contains 500 images of 2048$\times$1024 for validation. For all datasets, pixels belong to 19 shared semantic categories. During testing, we use mean intersection-over-union (mIoU) to evaluate the semantic segmentation performance.

**Implementation Details**. We use the DeepLab-V2 @deeplab with VGG16 @vgg backbone as the segmentation model. For the source training stage, following @adaptseg [@park2020discover], we use SGD with an initial learning rate $2.5\times10^{-4}$, momentum 0.9 and weight decay $5\times10^{-4}$ to optimize the model. For the target training stage, the learning rate is reduced to $1\times10^{-4}$. In both stages, we use the polynomial decay with a power of 0.9 to schedule the learning rate. The total training process takes 150K iterations, with a batch size of 1. We set $\tau$=0.9 and $q$%=50% for generating pseudo-labels. For CSPP, the number of patches $n$ and the activation probability $\beta$ are set to 4 and 0.3, respectively. By default, we use the inter-image CSPP and inject it after the first and second blocks of the VGG16. Note that, we use a batch size of 4 for CPSS, but optimize the model with only the first image. This can greatly reduce the computational cost, because using a batch size of 1 or 4 achieves a similar performance. All models are trained with one GTX 2080 TI GPU (11GB).

Comparison with State-of-the-Art Methods
----------------------------------------

<span>l|c|p<span>1cm</span>&lt;p<span>1cm</span>&lt;p<span>1cm</span>&lt;|p<span>1.3cm</span>&lt;|p<span>0.7cm</span>&lt;p<span>0.7cm</span>&lt;</span> Methods & Source & & Open(O) &\
GTA5 $\rightarrow$ & Free& Rainy & Snowy & Cloudy & Overcast & C & C+O\
Source Only & & 16.2 & 18.0 & 20.9 & 21.2 & 18.9 & 19.1\
AdaptSeg @adaptseg& & 20.2 & 21.2 & 23.8 & 25.1 & 22.1 & 22.5\
CBST @cbst& & 21.3 & 20.6 & 23.9 & 24.7 & 22.2 & 22.6\
IBN-Net @ibn& & 20.6 & 21.9 & 26.1 & 25.5 & 22.8 & 23.5\
PyCDA @pycda& & 21.7 & 22.3 & 25.9 & 25.4 & 23.3 & 23.8\
Liu  @liu2020open& & 22.0 & 22.9 & 27.0 & 27.9 & 24.5 & 25.0\
Park  @park2020discover& & 27.0 & 26.3 & 30.7 & 32.8 & 28.5 & 29.2\
Source Only& & 23.6 & 24.4 & 27.8 & 29.5 & 25.6 & 26.3\
AdaptSeg @adaptseg& & 25.6 & 27.2 & 31.8 & 32.1 & 28.8 & 29.2\
MOCDA @gong2020cluster& & 24.4 & 27.5 & 30.1 & 31.4 & 27.7 & 29.4\
Park  @park2020discover& & 27.1 & 30.4 & 35.5 & 36.1 & 32.0 & 32.3\
Ours (Stage-I)& & 28.5 & 30.5 & 36.4 & 37.4 & 32.8 & 33.2\
Ours (Stage-II)& & **30.6** &**31.9** & **37.6** & **38.0** & **34.4** & **34.5**\

**Results of GTA5 $\rightarrow$ C-Driving.** In Tab. \[tab:sota-GTA\], we compare our method with the state-of-the-art UDA models @adaptseg [@cbst; @ibn; @pycda] and OCDA models @liu2020open [@park2020discover; @gong2020cluster] on the setting of "GTA5 to C-Driving". For a fair comparison, all the models adopt DeepLab-V2 with VGG16 backbone. Following @park2020discover, we use the long training scheme (150K iterations) to train the model. We make the following observations. First, the models trained with the long training scheme produce higher results, showing the advantage of the long training scheme. Second, our "Stage-I" model, which is trained only with the source data, achieves the best performance among all the existing methods that use both the source and the target data. This verifies the effectiveness of the proposed CSPP in learning a generalizable model. Third, our "Stage-II" model outperforms all compared models by a large margin, indicating that our method produces new state-of-the-art performance for OCDA, even under the source-free constraint.

**Results of Domain Generalization.** We also verify the generalization ability of our method on CityScapes in Tab. \[tab:cityscapes\]. All models are trained with the VGG16 backbone. We can obtain the following findings. First, our "Stage-I" model surpasses the state-of-the-art domain generalization methods when training only with GTA5. Compared with DRPC @DRPC that additionally uses ImageNet @imagenet images, our model is 1.0% higher than it. Second, when training the model with GTA5 and (unlabeled) C-Driving, our "Stage-II" model significantly outperforms the compared methods, MOCDA @gong2020cluster and AdaptSeg @adaptseg by a large margin. These two findings demonstrate the effectiveness of the proposed method on open domains.

<span>0.48</span>

  Method                              GTA5 $\rightarrow$ CityScapes
  ---------------------------------- -------------------------------
  ASG @ASG                                        31.5
  IBN-Net @ibn                                    34.8
  DRPC @DRPC$\S$                                  36.1
  Ours (Stage-I)                                **37.1**
  MOCDA @gong2020cluster$\ddagger$                31.1
  AdaptSeg @adaptseg$\ddagger$                    32.8
  Ours (Stage-II)$\ddagger$                     **38.1**

  : Effectiveness of Style Augmentations.<span data-label="tab:style_aug"></span>

<span>0.48</span>

  Model    CPSS   PT      C         C+O
  ------- ------ ---- ---------- ----------
                         25.6       26.3
                         31.2       32.0
                       **32.8**   **33.2**
                         33.3       33.5
                         34.3       34.4
                       **34.4**   **34.5**

  : Effectiveness of Style Augmentations.<span data-label="tab:style_aug"></span>

Evaluation {#sec:ablation}
----------

**Effectiveness of Style Augmentations.** In Tab. \[tab:style\_aug\], we investigate the effectiveness of the proposed CPSS and photometric transformation (PT). Clearly, CPSS consistently improves the performance for both stages. Specifically, for the source training stage (Stage-I), inserting CPSS outperforms the baseline by 5.6% in C mIoU and by 5.7% in C+O mIoU. Adopting the photometric transformation further gains 1.6% and 1.2% improvement in C mIoU and C+O mIoU, respectively. For the target training stage (Stage-II), we initialize the model by the source model trained with CPSS and PT. Without using style augmentations, self-supervised learning achieves limited improvement. In contrast, adding CPSS can clearly promote the performance on both compound and open domains. This verifies that CPSS can not only reduce the impact of noisy samples but also improve the robustness of the model to unseen domains. On the other hand, using photometric transformation has a slight influence on the performance. This is mainly because that the model has been familiar with such transformation during source training.

**Comparison of Different Stylized Operations.** In Tab. \[tab:operations\], we compare several stylized operations that do not use any auxiliary information, , MixStyle @zhou2021mixstyle, CrossNorm @crossnorm, and two versions of our CPSS. Results are conducted on the source training stage. We can find that mixing styles with a random weight (MixStyle) is less suitable for semantic segmentation, because MixStyle may sometimes generate semantically unrealistic styles. Compared with CrossNorm and CPSS (intra-image), CPSS (inter-image) produces clearly higher performance. This indicates that augmenting samples with more various styles can help us to learn a more generalizable model in semantic segmentation.

**Is Splitting Latent Domains Necessary?** Recent OCDA methods @park2020discover [@gong2020cluster] show that the sub-domain labels can be used to reduce the latent domain gaps in the target domain. Instead, in our target training stage, we randomly select training samples from the target data to form the mini-batch without considering the sub-domain labels. To verify the impact of considering the latent domains for CPSS, we implement our framework with a new sampling strategy. Specifically, we sample the images in a balanced way, so that each mini-batch contains at least one sample for each sub-domain. We provide two kinds of latent domains: "Oracle" denotes using the original rainy, snowy, cloudy as the latent domains; and "Clustering" denotes separating latent domains by clustering the style features. As shown in Tab. \[tab:latent\], the random sampling strategy and its two variants achieve similar performance. This indicates that the proposed CPSS can potentially consider the style variations among different latent domains and learn a robust model, even without considering the factor of latent domains.

<span>0.55</span>

  Method                           C         C+O
  ---------------------------- ---------- ----------
  MixStyle @zhou2021mixstyle      30.7       31.2
  CrossNorm @crossnorm            31.4       31.8
  CPSS (intra-image)              31.7       32.3
  CPSS (inter-image )           **32.8**   **33.2**

  : Impact of latent domains.<span data-label="tab:latent"></span>

<span>0.44</span>

   W/ Latent     Split         C         C+O
  ----------- ------------ ---------- ----------
               Clustering   **34.4**   **34.7**
                 Oracle       34.3       34.5
                   —        **34.4**     34.5

  : Impact of latent domains.<span data-label="tab:latent"></span>

Parameter Analysis
------------------

We further analyze the sensitivities of CPSS to three important hyper-parameters, , the activation probability $\beta$, the number of patches $n$ and the injecting location $l$. By default, we vary the value of one parameter and keep the others fixed. Experiments are conducted in the source training stage.

**Patch Number $n$.** We compared the results of using different numbers of patches $n$ in Fig. \[fig:parameter\](a). When $n$=0, the model is trained without CPSS. With the increase of $n$, the model is encouraged to face more styles, producing higher results. However, when $n$ is too large, , 8, the patches are too small, which may generate less realistic styles and thus reduces the performance.

**Activation Probability $\beta$.** In Fig. \[fig:parameter\](b), we investigate the effect of the probability $\beta$ of activating the CPSS operation. The performance first increases with the value of $\beta$ and achieves the best results when $\beta$=0.3. However, assigning a larger value to $\beta$ (*e.g., 0.7*) leads to performance degradation. The results show that diversified styles can improve the generalization but training with excess generated styles fails to further improve the model performance.

**Injecting Location $l$.** In Fig. \[fig:parameter\](c), we estimate the impact of injecting CPSS into different blocks of the network. Block-$0$ denotes injecting the CPSS before the network, and block-$l~(l>0)$ denotes injecting CPSS before the last pooling layer of the $l$th convolutional block. We make two observations. First, injecting CPSS into shallow layers, , block-$0,1,2,3$, helps to improve the performance, while the performance degrades when injecting CPSS into a deep layer (block-$4,5$). The reason is that the mean and standard deviation represent style information in shallow layers but contain more semantic information in deep layers. Second, jointly injecting into multiple (two or three) layers can achieve further improvement. Considering the trade-off between accuracy and runtime, injecting CPSS into block-$1$ and block-$2$ is an appropriate choice.

![Sensitivities to (a) number of patches, (b) activation probability and (c) injecting location of CPSS.<span data-label="fig:parameter"></span>](figures/parameter_analysis.pdf){width="95.00000%"}

![Qualitative comparison of segmentation result on the compound domain (rainy, snowy, and cloudy) and open domains (overcast and CityScapes).<span data-label="fig:seg_results"></span>](figures/seg_results.pdf){width="95.00000%"}

![Examples of stylized images of CPSS. We directly apply CPSS on the image-level for image pairs on GTA5 (a and b) and C-Driving (c). The number of patches is set to 4.<span data-label="fig:cpss_visual"></span>](figures/cpss_visual.pdf){width="95.00000%"}

Visualization {#sec:visualization}
-------------

**Qualitative Comparison of Segmentation Results.** We compare the segmentation results for different models on the compound domain (rainy, snowy, cloudy) and open domains (overcast and CityScapes) in Fig. \[fig:seg\_results\]. Compared with the source only model and AdaptSeg @adaptseg, our models (Stage-I and Stage-II) clearly produce more accurate semantic results, especially for the boundaries between different objects. Comparing between our models, our Stage-II model can generate finer results on elements that have large intra-class variations between the virtual and real, *e.g.*, person, car and fence.

**Image-Level Visualization of CPSS.** To better understand the effect of our CPSS in style augmentation, we visualize three groups of style exchanging in Fig. \[fig:cpss\_visual\] by applying CPSS in the image-level (*i.e.*, block-$0$). For each group, we feed two original images (left column) into CPSS and generate corresponding stylized images (right column) by swapping patch styles among the 8 (2 $\times$ 4) patches. We obverse that the styles of patches are successfully changed and various patches are generated. We can easily infer that CPSS can also change styles in the feature-level. However, the style features may contain semantic information to a certain extent, which may cause image distortion during the style exchanging. These distorted results may hammer the model optimization. For future studies, CPSS could be developed to extract more style-specific features.

Conclusion
==========

In this work, we introduce a new setting, called source-free open compound domain adaptation (SF-OCDA), which has great potential in real-world applications. To address this problem, we propose an effective framework to train robust source and target models under the source-free constraint. Moreover, the Cross-Patch Style Swap (CPSS) is proposed to diversify the feature-level samples with various styles, consistently boosting the performance for both source and target training stages. Extensive experiments demonstrate the effectiveness of the proposed CPSS. Our method achieves state-of-the-art results on OCDA and DG benchmarks.

Appendix {#appendix .unnumbered}
========

Additional Experimental Results {#sec:add-experiment}
===============================

**Results of SYNTHIA $\rightarrow$ C-Driving**. In Tab. \[tab:sota-SY\], we compare our method with state-of-the-art methods on the setting of "SYNTHIA $\rightarrow$ C-Driving". Clearly, (1) the proposed method largely improves the performance of the source only model, and (2) our two models (Stage-I and Stage-II) both significantly outperform the state-of-the-art methods, verifying the generalization ability of the proposed method with different source datasets. We also find that the improvement of our Stage-II is limited. This is because given a poorly trained source model ($\approx$ 24% mIoU), we fail to generate enough useful / reliable pseudo-labels for self-supervised learning on the target domain. In our experiments, training the target model without the proposed CPSS will reduce the performance. This phenomenon can also be observed for Advent @advent, which additional uses entropy information to train the AdaptSeg @adaptseg but achieves lower results on C-Driving (in Tab. \[tab:sota-SY\]). In contrast, using our CPSS can alleviate the impact of wrong pseudo-labels and can guarantee that self-supervised learning will not hamper the model performance.

<span>l|c|p<span>1cm</span>&lt;p<span>1cm</span>&lt;p<span>1cm</span>&lt;|p<span>1.3cm</span>&lt;|p<span>0.7cm</span>&lt;p<span>0.7cm</span>&lt;</span> Methods & Source & & Open(O) &\
SYNTHIA $\rightarrow$& Free& Rainy & Snowy & Cloudy & Overcast & C & C+O\
Source Only @park2020discover& & 16.3 &18.8 &19.4 &19.5 &18.4 &18.5\
CBST @cbst& & 16.2 & 19.6 & 20.1 & 20.3 & 18.9 & 19.1\
CRST @crst& & 16.3 & 19.9 & 20.3 & 20.5 & 19.1 & 19.3\
AdaptSeg @adaptseg& & 17.0 & 20.5 & 21.6 & 21.6 & 20.0 & 20.2\
Advent @advent& & 17.7 & 19.9 & 20.2 & 20.5 & 19.3 & 19.6\
Park  @park2020discover& & 18.8 & 21.2 & 23.6 & 23.6 & 21.5 & 21.8\
Source Only$^\ast$& & 18.9 & 19.7 & 20.4 & 21.3 & 19.7 & 20.1\
Ours (Stage-I)& & **22.4**& 23.8 & **25.3** & **26.4** & 24.0 & 24.5\
Ours (Stage-II)& & **22.4**& **24.5** & **25.3** & **26.4** & **24.2** & **24.7**\

**Implementing AdaptSeg with CPSS**. To further demonstrate the generalization ability of the proposed CPSS, we inject CPSS into the widely used domain adaptation approach, AdaptSeg @adaptseg, and evaluate the results on the settings of "GTA5 $\rightarrow$ C-Driving" and "GTA5 $\rightarrow$ CityScapes". Note that, when using AdaptSeg, the source-free constraint is not enforced. Clearly, CPSS can consistently improve the performance of AdaptSeg by a large margin on both settings. This further confirms the compatibility of the proposed CPSS.

<span>l|c|p<span>0.8cm</span>&lt;p<span>0.8cm</span>&lt;p<span>0.8cm</span>&lt;|p<span>1.3cm</span>&lt;|p<span>0.5cm</span>&lt;p<span>0.5cm</span>&lt;|c</span> & & &\
& & & Open(O) & &\
& &Rainy & Snowy & Cloudy & Overcast & C & C+O &\
AdaptSeg @adaptseg & & — & — & — & — & — & — & 35.0\
AdaptSeg @adaptseg$^\ast$ & & 25.6 & 27.2 & 31.8 & 32.1 & 28.8 & 29.2 & 34.2\
AdaptSeg @adaptseg & & **28.9** & **29.1** & **35.2** & **36.0** & **31.9** & **32.3** & **38.5**\

**Per-Class IoU on GTA5 $\rightarrow$ C-Driving**. In Tab. \[tab:gtaperclass\_target\], we report the per-class IoU on different sub-domains of "GTA5$\rightarrow$C-Driving". Generally, our methods (Stage-I and Stage-II) produce higher results on most classes for all sub-domains. On the other hand, we find that all the methods fail to recognize the samples of the "train", "motorcycle" and ""bicycle" classes, which are rarely appeared in the C-Driving dataset.

Comparison of Different Stylized Operations {#sec:operations}
===========================================

The proposed Cross-Patch Style Swap (CPSS) is closely related to MixStyle @zhou2021mixstyle and CrossNorm @crossnorm, which are both designed for domain generalization. All three methods aim to improve the generalization ability of the model by perturbing style features of training samples. However, the stylized operations of them are different. Specifically, MixStyle replaces the style of a sample with the one that is generated by mixing its own style feature with a shuffled style feature using a random convex weight. Instead, CrossNorm directly exchanges the styles of two samples, which is a special case of MixStyle when the weight of the shuffled style feature is 1. Both MixStyle and CrossNorm compute one style feature for each sample and stylize each sample with one style feature. Different from them, our CPSS generates several styles for each sample by separating the feature map into different patches. This modification is specially designed for semantic segmentation in the self-driving scenario, because patches in a frame could contain different styles. Compared with MixStyle and CrossNorm, our CPSS can provide more diverse and useful styles for generating stylized feature maps. In addition, with CPSS, the model is trained with richer feature maps where each one jointly contains multiple different styles, further enforcing the model to be robust to style variations. In our main paper, we conduct experiments by comparing these three methods and show that CPSS produces clearly higher results than the other two methods.

[^1]: Equal contribution.

{'source': 'arXiv_out/2106'}