Generalizable Model-agnostic Semantic Segmentation via Target-specific Normalization
Jian Zhang Lei Qi Yinghuan Shi Yang Gao
cs.CV






nju]Jian Zhang 
seu]Lei Qicor
nju]Yinghuan Shicor
nju]Yang Gao

[cor]Corresponding author: Yinghuan Shi (syh@nju.edu.cn) and Lei Qi (qilei@seu.edu.cn)

[nju]State Key Laboratory for Novel Software Technology, Nanjing University, China
[seu]Key Lab of Computer Network and Information Integration (Ministry of Education), Southeast University, China.


    Semantic segmentation in a supervised learning manner has achieved significant progress in recent years. However, its performance usually drops dramatically due to the data-distribution discrepancy between seen and unseen domains when we directly deploy the trained model to segment the images of unseen (or new coming) domains. To this end, we propose a novel domain generalization framework for the generalizable semantic segmentation task, which enhances the generalization ability of the model from two different views, including the training paradigm and the test strategy. Concretely, we exploit the model-agnostic learning to simulate the domain shift problem, which deals with the domain generalization from the training scheme perspective. Besides, considering the data-distribution discrepancy between seen source and unseen target domains, we develop the target-specific normalization scheme to enhance the generalization ability. Furthermore, when images come one by one in the test stage, we design the image-based memory bank (Image Bank in short) with style-based selection policy to select similar images to obtain more accurate statistics of normalization. Extensive experiments highlight that the proposed method produces state-of-the-art performance for the domain generalization of semantic segmentation on multiple benchmark segmentation datasets, , Cityscapes, Mapillary. 
    
    
    
    


Domain generalizationSemantic segmentationModel-agnostic learningTarget-specific normalization









§ INTRODUCTION


Semantic segmentation is a long-standing yet challenging task in computer vision and pattern recognition community, which aims to assign the semantic label (, person or car) to each pixel in a given image. Currently, in real-world tasks, semantic segmentation has a wide range of applications, such as autonomous driving, security surveillance, and augmented reality. 
Recently, thanks to deep convolutional neural networks, we have witnessed the significant breakthrough of semantic segmentation on various benchmark datasets.
Those powerful methods perform well in an identical distribution assumption between training and test images in the traditional supervised setting – the training and test images are collected from the same domain. Despite their success, when this assumption could not satisfy in the real case, , when the training and test images come from different domains, the performance drops dramatically.


Therefore, unsupervised domain adaptation (UDA) <cit.> is proposed to deal with this issue. In the setting of UDA, the model was trained with 1) labeled data from the source domain and 2) unlabeled data from the target domain. Basically, the common goal of UDA methods is to mitigate the domain shift between the source domain and the target domain. In this meaning, the model is expected to generalize well in the target domain. However, when applying the trained UDA segmentation model to other unseen domains, it still requires to first collect unlabeled data from new domains and then retrain the model with the newly collected data. In this case, the burden from two sides, , data collection and model retraining, is inevitable.

Notably, we expect that one model can be trained only once yet could generalize well in other unseen (or new coming) scenarios without any additional retraining procedure. With this expectation, deployment of the trained model to unseen scenarios becomes feasible, since the cost for data collection, data annotation, and model training can be simultaneously reduced. This setting is referred to as domain generalization (DG) that considers how to acquire knowledge from an arbitrary number of related domains, and then apply the trained model directly to segment/classify previously unseen domains <cit.>. The key difference between UDA and DG setting relies on whether target domain data could be observed and used in the training stage, as shown in Fig. <ref>. Although DG has a similar goal with UDA, it is much more challenging since we have no available samples in the target domain. That is to say, the target domain in DG cannot be used, or even observed.



Recently, DG for image classification task has attracted lots of attention <cit.>. However, according to our best knowledge, only very few methods focus on the setting of DG in semantic segmentation (, generalizable semantic segmentation). For example, IBN-Net <cit.> exploits IN to alleviate the style discrepancy across different domains. Also, the method in <cit.> transfers the synthetic images to real images with auxiliary datasets and learns the domain-invariant representations with augmented dataset effectively. 
We notice that these methods all utilize the plain training paradigm, which is easy to overfit the source domains. Besides, during the test stage, the statistics of the normalization are from the source domain, which could suffer from the domain shift. 





Different from these aforementioned methods, we address domain generalization of semantic segmentation from 1) training paradigm and 2) test strategy perspectives together. We put forward a novel domain generalization framework to help model well generalize to the unseen target domain in semantic segmentation.

In particular, for the training paradigm, we adopt the model-agnostic learning method (, meta-learning for domain generalization (MLDG) <cit.>) to simulate the domain shift problem with episodic training paradigm, which has demonstrated its superior performance in image classification. It can be regarded as a regularization mechanism that prevents the model from overfitting the source domains.


For the test strategy, we propose target-specific normalization to alleviate the data-distribution discrepancy between seen source and unseen target domains in the test stage. Specifically, regardless of training CNN in supervised, semi-supervised, unsupervised, or even the domain adaptation setting, we often use the mean and variance calculated from the training samples to conduct the batch normalization in the test stage. Since the distribution of source domains is different from the unseen domains, normalizing the unseen domains with these statistics could suffer from the domain shift problem. To address this issue in the DG setting, we utilize the mean and variance calculated from the target domain during the test procedure to perform normalization, which can adapt to various unseen domains without any finetuning step on the network.



However, the inaccurate estimation of statistics in batch normalization might degrade the performance, which is caused by the small batch size (especially for the batch size of 1 in the test stage of DG). 
To boost the performance of the network with an accurate estimation, we propose to store previously tested images as an image-based memory bank (abbreviated as Image Bank in the following part). When test images come one by one, we select images from the Image Bank for an accurate statistics estimation of the current sample with style-based selection policy. 

Extensive experiments demonstrate that the proposed method produces state-of-the-art performance for the domain generalization of semantic segmentation on multiple benchmark segmentation datasets, , our method improves up to 5.54% based on a strong baseline. Moreover, according to our ablation study, we sufficiently validate the effectiveness of each component in the proposed method.

Our contributions can be summarized as follows:

  
  
  * We develop a novel domain generalization framework that jointly exploits the model-agnostic training scheme and the target-specific normalization test strategy to address the generalizable semantic segmentation task.
  
  * To deal with the inaccurate statistics estimation for a single sample in the test stage, we propose to utilize the Image Bank with the style-based selection policy to obtain accurate statistics.
  
  
  * We provide a strong baseline for semantic segmentation DG problem. Moreover, the proposed method achieves state-of-the-art performance for DG of semantic segmentation on multiple benchmark datasets. 



§ RELATED WORK




We review the previous literature from the related topics, , unsupervised domain adaptation, domain generalization, and normalization. 

Semantic segmentation. Semantic segmentation belongs to a pixel-level classification task which is a fundamental yet challenge problem in computer vision and pattern recognition field. For accurate segmentation, several works aggregate multi-scale contextual information <cit.> and some of them refine low-resolution feature maps with high-resolution feature maps. Recently, more attention has been paid to the long-range dependency <cit.> between pixels for better segmentation. Although these methods can achieve promising segmentation accuracy in supervised setting when training and test domains belong to the same domain, how to generalize well to other unseen domains in semantic segmentation is still an open problem.

Unsupervised domain adaption. Domain adaptation (DA) is a particular case of transfer learning. It leverages labeled data in one or more related source domains to learn a classifier for the target domain. Among these DA settings, the setting when the labeled data is not available in the target domain is referred to as unsupervised domain adaptation (UDA), which has been received considerable attention recently. Some methods first align distributions in pixel-level space <cit.>, and then adopt a generative network to narrow the distribution gap between source and target domains. Chang <cit.> aligns the feature distribution between source and target domains by adversarial training to keep semantic features consistent in different domains. Differently, Sun <cit.> weights the loss function by paying more attention to regions with similar label structures. However, since there is no available data in target domains, we cannot directly align the distribution of source and target domains in our generalizable semantic segmentation task. 
Besides, some test time adaptation methods <cit.> are proposed to deal with the domain shift problem in test time. During the test procedure, these methods and our method aim to better exploit the data distribution in the target domain, but there is an obvious diﬀerence that these methods all need to train the model with back-propagation in the test stage, while our method only does the inference without the cost of training model. Thus, the deployment of these methods is limited.

Domain generalization. Different from UDA, domain generalization (DG) is a more challenging setting, which cannot use or even observe any target images during the training process. Current DG methods can be roughly classified into three categories: data augmentation, domain invariant feature learning and regularization. 

The augmentation-based methods try to augment source domains with diverse styles. The model trained with these augmented data can be robust to the unseen domains with unseen styles. The techniques of augmentation are mainly based on Generative Adversarial Networks (GANs <cit.>) <cit.> or Adaptive Instance Normalization (, AdaIN <cit.>) <cit.>. One of the segmentation method <cit.> in DG adopts CycleGAN to generate novel samples with image content from source domains and style from ImageNet.

The domain invariant feature learning-based methods try to learn an invariant representation across source domains. The representation should be able to generalize well to the unseen domain via the invariant hypothesis. All these methods try to first align the distribution by adversarial training <cit.> or learning domain-shared and domain-specific components explicitly. Then they classify the unseen sample with only domain-shared component <cit.>.

Regularization-based methods assume the overfitting problem largely hurts the generalization ability. Thus they aim to reduce the overfitting problem during training via the regularization mechanism. Carlucci  <cit.> adopt self-supervised learning as a regularization term to help the generalization. Huang  <cit.> propose to dropout the largest gradient during training to encourage distributed features and prevent overfitting. Meta-learning <cit.> based methods <cit.> can also be seen as an approach of regularization due to its episodic training mechanism. It imitates the DG procedure where the model trained in seen domains should perform well in unseen domains. It has been applied to the DG classification task successfully.  For example, methods in <cit.> are proposed to add a parameterized regularization term in meta-test stage to prevent the model from overfitting, while MLDG <cit.> directly adopts the MAML <cit.> training framework. Although these DG methods performs well on the classification task, few works are developed to solve the generalizable semantic segmentation task. 



Normalization. Batch normalization (BN) <cit.> and instance normalization (IN) <cit.> have been popularly used to stabilize training procedure and ease gradient explosion and vanishing problem. The literature <cit.> shows that BN can preserve discriminative information in a domain, while IN is effective in reducing appearance difference and feature divergence <cit.>. Thus, recent approaches attempt to learn independent BN for each domain <cit.> and employ IN <cit.> to alleviate the domain gap. Besides, the method in <cit.> leverages both BN and IN to exploit their advantages. However, all the above methods use the accumulated statistics (, mean and variance) in source domains to normalize the images from target domains, which could suffer from the domain-shift problem in DG due to no available data in the target domain. Recently, AdaBN <cit.> is proposed in UDA to utilize the statistics of the target domain to normalize images from the target domain. However, AdaBN calculates the statistics from all the test samples, which is unavailable in DG, while our method only requires a single image in the test stage.





§ OUR METHOD

In this section, we introduce the technical details of our method which consists of 1) the model-agnostic learning scheme in the training stage, 2) the target-specific normalization and 3) the Image Bank in the test stage, as illustrated in Fig. <ref>. In the following part, we first introduce notations in DG and present a simple method as a DG semantic segmentation baseline. We then formulate the meta-learning algorithm framework for semantic segmentation. Afterwards, we describe the target-specific normalization in the test stage. Lastly, we detail the Image Bank in the test stage.



 §.§ Our Baseline

Let 𝒳 denote the image space and 𝒴 denote the label space.
A domain is defined by a set of image and label pairs 𝒟^i = {(x_j^i, y_j^i)}_j=0^N_i, where x_j^i ∈𝒳, y_j^i ∈𝒴, and N_i is the number of samples in the domain 𝒟^i.
Assume that we have N domains. N-1 domains are source domains 𝒟_src = {𝒟^i}^N-1_i=1 and the left one domain is the target domain 𝒟_target = {𝒟^N}. DG methods train the network with 𝒟_src and test in 𝒟_target. 

Suppose we have a network to be trained. It contains a feature extractor F(x;θ) parametrized by θ and a classifier T(f;ϕ) parametrized by ϕ. For image semantic segmentation, we usually use the cross-entropy (CE) as loss function, which can be defined over dataset as follows:


    ℒ(𝒟; θ, ϕ) = 1/|𝒟|∑_(x, y)∈𝒟ℒ_CE(T(F(x;θ); ϕ), y).








A straightforward DG method called AGG is introduced to train the model with 𝒟_src directly. It means simply aggregating all training data. We expect to minimize the following cross-entropy loss in Eq. (<ref>).


    ℒ_agg = ℒ(𝒟_src; θ, ϕ).





Note that AGG is a promising baseline in the semantic segmentation of domain generalization when 𝒟_src consists of multiple source domains. We will show the experimental results in Section <ref>.



 §.§ Model-agnostic Meta-learning in DG


We can further improve the above baseline (, AGG) by using the model-agnostic learning scheme. Concretely, we adopt MLDG (, meta-learning for domain generalization) <cit.> to train our model. MLDG follows the MAML framework and has been successfully utilized in the DG classification task. In MAML, we construct many tasks from the auxiliary dataset, and each task contains a support set and query set. We hope that the network fine-tuned with the support set can generalize well in the query set. Similar to MAML, the DG task expects that the network fine-tuned in seen domains can generalize well in unseen domains. Thus, at each iteration, we randomly sample a task consisting of a sample set ℬ from each source domain. We partition ℬ into the meta-train data ℬ_tr and meta-test data ℬ_te, where ℬ_tr∩ℬ_te=∅, ℬ_tr∪ℬ_te=ℬ. These two sets have no intersection in domains and correspond to the support set and query set, respectively. We first calculate the domain-specific loss ℒ_ds by 


    ℒ_ds = ℒ(ℬ_tr; θ, ϕ).




Basically, we can calculate the gradient ▿ℒ_ds of ℒ_ds with back-propagation. Then the new parameter θ' and ϕ' can be updated with SGD (, stochastic gradient decent) as follows:
  
    θ'    = θ -  η * ▿_θℒ_ds;
    ϕ'    = ϕ -  η * ▿_ϕℒ_ds,

where η is the inner learning rate during model updating. Similar to MAML, we expect the updated network can generalize well in unseen domain. In this way, the domain generalization loss ℒ_dg is calculated with meta-test data 𝒟_te as follows:

    ℒ_dg = ℒ(ℬ_te; θ', ϕ').

We use both these two losses to update the origin parameters θ and ϕ. In this meaning, the network is expected to perform well in both seen and unseen domains. Formally, the final meta-learning loss is

    ℒ_meta = ℒ_dg + αℒ_ds,




where α is a weight parameter to balance these two terms. The whole network is updated with ℒ_meta and outer learning rate γ as described in Algorithm <ref>.







 §.§ Target-specific Normalization


In the conventional supervised, semi-supervised, unsupervised and domain adaptation scenarios, we normally use the accumulated mean μ̂∈ℝ^C and variance σ̂^2  ∈ℝ^C in the training stage to conduct normalization for the test data x ∈ℝ^N× C × H × W as follows:


    x̂_n,c,h,w = x_n,c,h,w - μ̂_c/√(σ̂_c^2 + ϵ)w_c + b_c.

N, C, H, W are number of images,  number of channels, image height and image width, respectively. x_n,c,h,w denotes the n× c× w× h-th element. 
However, in our DG setting, the statistics in source domains is different from target domain due to the domain gap. Considering this fact, we propose the target-specific normalization (TN) to directly use the statistics of target domain to normalize the features. Concretely, we obtain the new mean and variance for a mini-batch with M samples in the test stage as follows:

 
    μ̅_c    = 1/MHW∑_n=1^M∑_h=1^H∑_w=1^W x_n,c,w,h;
    σ̅_c^2    = 1/MHW∑_n=1^M∑_h=1^H∑_w=1^W (x_n,c,w,h-μ̅_c)^2.


  Therefore, in the test stage, Eq. (<ref>) can be rewritten as below:
  
    x̂_n,c,h,w = x_n,c,h,w - μ̅_c/√(σ̅_c^2 + ϵ)w_c + b_c.



Remark. For the target-specific normalization (TN), we present some analysis and observations as follows: 1) In our experiments, we find that if we merely use the target-specific normalization on a model trained by the standard training method (, non-MLDG training scheme), TN cannot effectively improve the generalization ability of the model in the unseen target domain; 

2) The size of a mini-batch has an impact on the final performance.  Besides, if M>1, the different sample combinations in a mini-batch might obtain different results. To address this issue, each experiment is repeated 5 times, and the average results are reported. 







 §.§ Image Bank

In real-world applications, several test images could come one by one. We now provide an example about online image recognition for illustration: to predict the label for a given query image, if we deploy our trained model on a website, the user can merely upload one image to the website. In this situation, we have to process the uploaded images one by one for each user (, batch size is 1 in test stage). Although the target-specific normalization can adapt the model to the unseen domain with statistics calculated in the test stage, we argue that it cannot produce accurate statistics estimation for a single image in the test stage. To obtain more accurate statistics, we put previously tested images into an Image Bank to aid the estimation of new coming images. Specifically, we adopt Image Bank with a fixed size of Q to store these images. When an unseen image comes, we combine it with several previously tested images to produce its statistics. 
We now introduce our style-based selection policy in our Image Bank.

Style-based selection policy. We notice that simply utilizing the previously stored images in the Image Bank could suffer from the domain shift problem when images in the Image Bank are from different domains. It might also result in an inaccurate estimation. The images from similar domains have similar style <cit.>. and data distribution. And they can help robust statistics estimation. Being aware of this fact, we propose the Style-based selection policy for Image Bank (SIB) to select the images that are similar to the current image in the style, as illustrated in Fig. <ref>. Specifically, as aforementioned, since the normalization statistics represent the style (distribution) information <cit.>, we adopt s=(μ, σ) in the statistics to represent the style information in a given image. Considering the lower layers capture more detailed style information, to better represent the style information, we first extract the style statistics from the first layer of ResNet. Then, we calculate the symmetric KL Divergence <cit.> between styles of the current image and the images in the Image Bank as follows:

    D(s_i || s_j)     = KL(s_i||s_j) + KL(s_j||s_i); 
    
        KL(s_i || s_j)    = logσ_j/σ_i + σ_i^2+(μ_i-μ_j)^2/2σ^2_j-1/2.

Finally, we select the top M similar images from Image Bank and concatenate them with the current image to generate the statistics for the current image. When the Image Bank is full, we simply remove the out-of-date images with First In, First Out (FIFO) strategy. We also employ the random removal strategy and find no obvious difference in performance. Since the order in which the test images arrive is random, FIFO could equal to random removal scheme.




§ EXPERIMENTS

In this section, we report both the quantitative and qualitative results. Specifically, we first describe the detail of datasets and implementation. Then, we extensively compare our method with state-of-the-art DG and UDA methods. Moreover, we conduct an ablation study to confirm the effectiveness of each module used in our framework. Lastly, we analyze the properties of our method systematically.



 §.§ Dataset and Implementation Details


Dataset. We totally introduced five semantic segmentation datasets including: Cityscapes <cit.>, GTA5 <cit.>, Synthia <cit.>, IDD <cit.> and Mapillary <cit.> for the evaluation. 
Note that all these five datasets have the same class space (, they have the same 19 classes). For the simplicity, the five domains are denoted as G, S, I, M, C for GTA5, Synthia, IDD, Mapillary and Cityscapes, respectively, in the following parts. The detail of each dataset is as follows:


    
  * Cityscapes (C) is a large-scale real-world street scene dataset consisting of 2,975 training, 500 validation and 1,525 test images, respectively. All the images are high resolution with a size of 2048 × 1024.
    
  * GTA5 (G) is collected from the GTA5 game, resulting in accurate pixel-wise semantic labels. There are 24,966 images with a size of 1914 × 1052. 
    
  * Synthia (S) comprises synthesized images that are generated by rendering a virtual city. We use a subset of this dataset called SYNTHIA-RAND-CITYSCAPES, including 9,400 images with the resolution of 1280 × 760.
    
  * IDD (I) is more diverse compared with cityscapes. A total of 10,004 images are captured from Indian roads with the resolution of 1678×968.
    
  * Mapillary (M) is regarded as the largest dataset with 25,000 high-resolution images. The images are collected from all over the world and taken from a diverse source of image capturing devices. The dataset contains 18,000 training images, 2,000 validation images, and 5,000 test images, respectively. The size of all these images is larger than 1920 × 2080.



Implementation details.
 We use ResNet-50 <cit.> pretrained on ImageNet <cit.> in this work. Following the conventional semantic segmentation setting, we use a dilated version of ResNet, resulting in an output stride of 8. 
 The inner and outer learning rates of η, γ are set to 1e-3 and 5e-3, respectively. The weight α is set to 1. 
 We adopt the SGD optimizer, where the weight decay and momentum are set to 0.9 and 5e-4, respectively. We employ the Poly scheduler to decrease the outer learning rate with a weight of 0.9. 
 We adopt extensive online data augmentation methods, including random flip, random scales at the range of [0.5, 2.0], random Gaussian blur, and random crop with a size of 600 × 600. We use mean IoU as our evaluation metric. 
 The total training epoch is 120. At each iteration, the images are randomly sampled from each domain, forming the image size of B× D × C × H × W, where B, D, C, H, and W are batch size, the number of domains, channels, height, and width, respectively. We use a batch size of 8 for training. The test batch size M and queue size Q are set to 4 and 128 respectively. When applying target-specific normalization, due to the variance of statistics in a mini-batch, all results are averaged in five runs with different random seeds in the test stage. We evaluate our method in the validation set if not mentioned. 
 


 §.§ Comparison with the States-of-the-arts




We compare our method with two state-of-the-art DG segmentation methods <cit.>. 
These two methods use GTA5 as the source domain and Cityscapes as the target domain. IBN-Net <cit.> plugs instance normalization into ResNet to remove style information. The method in <cit.> augments GTA5 dataset with various styles transferred from ImageNet. Thus the performance in <cit.> is better due to the larger dataset. For our method, we conduct experiments in two settings. The first one is using a similar setting to the literature <cit.>. The second one is using GTA5, Synthia, Mapillary, and IDD as source domains, , the “G+S+I+M → C” task. We evaluate our method in the test set of Cityscapes, as reported in Table <ref>. Note that “AGG” in this table denotes the baseline model. It uses the ResNet-50 as the backbone and is trained only in the source domain. “AGG-4” in Table <ref> indicates the baseline in the second setting. Firstly, in the same setting, our method can significantly outperform the method in <cit.>. For example, based on the higher baseline (, (36.94 vs. 32.45)), our method can still obtain the more improvement (, (5.97 vs. 4.97)). Secondly, in the “G+S+I+M → C” task, we gain a strong baseline. Moreover, the result can be further risen by 5.54% (61.19 vs. 55.65) on Cityscapes, which shows the effectiveness of our method. Note that our method also does not use the test data. When applying target-specific normalization, we only utilize previously seen test images instead of the whole test data to estimate the statistics. Therefore, the comparison is fair.






 §.§ Comparison with Unsupervised Domain Adaptation Methods
 

To further exhibit the superiority of our method, we also compare our method with some unsupervised domain adaptation methods. The results are reported in Table <ref>, which is obtained by submitting the predictions in the test set of Cityscapes. All these UDA methods are trained with labeled GTA5 or Synthia dataset and unlabeled Cityscapes dataset. To compare with them, due to the limitation of our methods that the number of source domains should bigger than 1, we train our model with both GTA5 and Synthia. As seen in Table <ref>, our method can achieve comparable performance to those UDA methods. Note that our method does not employ any data from the target domain to train the model. This disadvantage further highlights the superiority of our method. Also, our method is more valuable than the UDA methods in the real-world application, because we do not need to collect any data from a new scenario (, the unseen target domain).










 §.§ Ablation Study


To analyze the efficacy of each component, we perform ablation study on three and four source domains. We report the experimental results in Table <ref>. Firstly, compared with “AGG”, “MLDG” can obtain better performance, especially for the model trained on four source domains. For example, for the “G+S+I+M → C”  task, we achieve improvement by 2.24% (55.95 vs. 53.71). Thus, this confirms the effectiveness of MLDG in DG semantic segmentation. Secondly, we observe that the target-specific normalization benefits from the model trained by the model-agnostic learning scheme. Specifically, in the “G+S+I+M → C” task, “MLDG+TN” improves “MLDG” by 1.24% (55.95 vs. 57.19), while “AGG+TN” has an inferior performance than “AGG”. This comparison demonstrates that using the model-agnostic learning skill and target-specific normalization together can achieve good performance in the generalizable semantic segmentation task. Finally, the results of “MLDG+SIB” outperform “MLDG+TN” in all tasks. It validates that the target-specific module with Image Bank can enhance the ability of generalization in the unseen target domain. Besides, Our method has stable results with small std (, in the “G+S+M+I→C" task, the mean and std of our method are 57.66 ± 0.17), because the selection procedure in Image Bank tends to select most similar images, which can produce more stable statistics for each test image. Some visual results from the model trained on four source domains are shown in Fig.<ref>. As seen in the white bounding box, the segmentation results of our method are closer to the ground truth when compared to the AGG.






 §.§ Further Analysis


In this paragraph, we conduct more experiments to further analyze the property of the proposed method from multiple different views. 

Influence of the divided ratio of meta-train and meta-test. In MLDG, we need to split source domains into meta-train and meta-test domains, , we can split four source domains into three meta-train domains and one meta-test domain or one meta-train domain and three meta-test domains. To investigate the best option to split the four source domains, we enumerate the partition ratio of meta-train and meta-test, including “3:1”, “2:2”, and “1:3”. The performance is shown in Table <ref>. As seen, the best option is to split the source domains into “2:2”. We hypothesize that the balanced partition scheme guarantees that both of the two models trained on mete-train and meta-test domains can achieve relatively good performance, thus the final model can better generalize in unseen domains. For all the experiments with four source domains in this paper, we set meta-train and meta-test to 2 and 2. Besides, the meta-test procedure is critically important when comparing the results of the “3:1” to “1:3” (, using three source domains as meta-test produces better outcome), which imitates the process of domain generalization in the training stage. This meta-test step is also mainly different from the general training method that the model is directly trained using all source domains (, the baseline, AGG).

 

Evaluation of the batch size in target-specific normalization.  In SIB, the batch size M has an impact on the mean and variance. Thus we utilize different batch sizes to conduct the experiment and analyze the influence of different batch sizes in the test stage. The experimental results are illustrated in Fig. <ref>.
As seen in this figure, with the increasing batch size, the performance gradually improves on the two models trained by AGG and MLDG. 
Note that An obvious trade-off is shown. When the batch size is increasing, the performance first increases and achieves the best performance, then decreases a little afterward. We hypothesize that there is a balance between specificity and commonality which corresponds to the image-specific style and domain-specific style in an image. When the batch size is 1, the statistics are solely based on the single image which is highly related to the image-specific style. When the batch size is large enough, the statistics are averaged over many images, which removes the specificity of the current image and is most related to the domain-specific style. 



Besides, as seen from Fig. <ref>, although target-specific statistics increases the performance of the AGG model of batch size from 1 to 16, the result drops drastically when compared with the baseline of our method which uses the statistics from the training stage (, the conventional batch normalization). Furthermore, its improvement is also little when the batch size is small. Differently, when we apply target-specific statistics on the MLDG model, even the batch size of 1 can still be better than the baseline. This further demonstrates that MLDG can better show the superiority of the target-specific normalization in the generalizable semantic segmentation task. Therefore, leveraging them together is excellently critical in the proposed method. 






The effectiveness of style-based selection policy. When images come one by one, to better address the discrepancy of image distribution problem in the Image Bank, the Style-based selection policy for Image Bank (SIB) is proposed to estimate the statistics with images from similar domains in the Image Bank. We evaluate its performance on “G+S+I → C+M" task. The simple Queue-based Image Bank is denoted as QIB. As shown in Table <ref>, when we use QIB, the performance drops comparing to “MLDG+TN” due to the inaccurate statistics estimated with images from multiple domains in the Image Bank. After we apply SIB, images from the same domain are selected and the accurate statistics produce better performance. In particular, if test images come from the same target domain, the image bank based method equals to the target-special normalization with multiple images in a batch, as shown in Fig. <ref>.



 
 

Influence of different layers to capture the style information. 
Although SIB can effectively deal with the inaccurate estimation of statistics of multi-domain images. The quality of the similarity in SIB is highly dependent on using which layer in Resnet to obtain the style information. We conduct an experiment to investigate its influence. The mean IoU and the accuracy of selecting images from the same domain are all reported in Table <ref>. It is observed that the best accuracy and segmentation performance are achieved when we choose the first layer to obtain the style information. Since it contains detailed style information, it is more accurate to calculate the similarity of styles between images.






  Evaluation of queue size in Style-based Image Bank. Besides that layer choice can affect the performance of SIB, the queue size also has an influence. As shown in Table <ref>, with the queue size increases, the performance first increases and then decreases. We hypothesize that with a larger queue size, more images from the same domain can be selected for an accurate estimation of statistics. However, when selected images are too similar, the estimation will also be inaccurate because of little diversity in these images. Thus a relatively large queue size (, 128) can strike a balance.
  

 
Influence on the number of source domains. The number of source domains has a significant impact on the performance of DG. We conduct some experiments using a different number of source domains and utilize Cityscapes as the target domain. The results are reported in Table <ref>. As seen, more source domains can achieve better performance. Besides, we observe that if the source domains have a similar domain with the target domain, the DG method can obtain significant improvement. For example, images from both GTA5 (G) and Synthia (S) are synthetic, and the images from IDD (I) and Mapillary (M) are collected from the real scenario, which are closer to the real dataset Cityscapes (C). Thus, the outcome from “G+S → C” to “G+S+I → C” has a larger advance than the performance from “G → C” to “G+S → C”.






Analysis of meta-learning and target-specific normalization. To further give prominence to the property of our method, we analyze the experimental results from the source domain perspective. We report the results in Table <ref>. According to the comparison between the top group (“AGG” and “AGG+SIB”) and the bottom group (“MLDG” and “MLDG+SIB” (Our Method)), the target-specific normalization cannot only enhance the generalizable ability in the unseen target domain but also improve the performance in source domains on the MLDG model. However, when we utilize SIB on the AGG model, the performance has a large deterioration in source domains. We suppose that the AGG model is sensitive to the statistics, while the model-agnostic scheme can guide the model to be more robust to it.


 
 
 
  





Effectiveness of our method in different backbones. To demonstrate that our method can easily be applied to different backbones, we have added the comparison to DeepLabV3 <cit.> and HRNet <cit.>. From the Table <ref>, we observe that using a stronger network can enhance the generalization ability of our method and our method can consistently improve based on other backbones (, our method improves MLDG by 2.35% (64.09 vs. 61.74) based on HRNet). 



Comparison to AdaBN. We have compared our method to AdaBN <cit.> in Table <ref>. Our method and AdaBN are all performed based on MLDG for a fair comparison. As reported from the table, our method can consistently outperform AdaBN on different tasks. As we have discussed in Section <ref>, each image has its specificity and commonality in its corresponding domain. To normalize the image, there is a balance between specificity (small batch size) and commonality (large batch size). When the batch size is 1, the BN statistics estimation is solely based on the single image which cannot produce accurate statistics for its corresponding domain. When the batch size is large enough, the specificity in the statistics could be removed. Thus, our method can strike a balance between them and perform better than AdaBN. Particularly, AdaBN requires to collect all data from the target domain to do normalization, which is impractical in DG when only one image is provided at each moment in the test stage.

Inference time of the target-speciﬁc normalization. We report the time cost on Tesla V100 GPU with Resnet-50 as the backbone and input image size of 769 × 769. Since the semantic segmentation task is a time-consuming task due to the large output size, the normal test speed (without SIB) is 0.12 seconds per image and when we apply SIB, the speed becomes 0.34 seconds per image. Although our method needs more time than the normal test, it can produce better performance (, in the “G+S+M+I→C" task, our method achieves 57.66 mIoU, which is better than 55.95 mIoU of the AGG method).

Discussion: In this paper, we consider two image processing scenarios including “collect-collect-...-collect-process” and “collect-process-collect” in the test stage. When there are several test images available one-by-one (, “collect-process-collect” scenario), the proposed Style-based selection policy of Image Bank (SIB) can calculate more accurate statistics and perform better than the target-specific normalization (TN) designed for handling the “collect-collect-...-collect-process”  case, as shown in Table <ref>. Besides, compared with the typical batch normalization (BN), both our SIB and TN can achieve better performance in the “one by one” case.



§ CONCLUSION

In this paper, we developed a novel domain generalization method to handle the generalizable semantic segmentation task, which jointly exploits the meta-learning training skill and the target-specific normalization from the model training and the data-distribution discrepancy perspectives. To deal with the inaccurate estimation of statistics in target-specific normalization, the Image Bank with the style-based selection policy was designed for an accurate estimation. By extensive evaluation, the efficacy of the proposed method was thoroughly validated from multiple different views. Besides, we got an interesting observation that the model trained by the model-agnostic learning skill can better highlight the advantage of the target-specific normalization in semantic segmentation of domain generalization. In future work, we will further explore the reason for the above observation and develop a better solution for the DG semantic segmentation. 

Acknowledgement.
This work was supported by the National Key Research and Development Program of China (2019YFC0118300), China Postdoctoral Science Foundation funded project (2021M690609) and Jiangsu Natural Science Foundation (BK20210224).

















