Domain Adaptation Meets Disentangled Representation Learning and Style Transfer
Hoang Tran Vu Ching-Chun Huang
cs.CV

headings


18SubNumber801  


 
Domain Adapt. Meets Disentangled Represent. Learning and Style Transfer
Hoang Tran Vu and Ching-Chun Huang


Department of Electrical Engineering, National Chung Cheng University, Taiwan

Domain Adaptation Meets Disentangled Representation Learning and Style Transfer
    Hoang Tran Vu and Ching-Chun Huang
    December 30, 2023
===============================================================================





Many methods have been proposed to solve the domain adaptation problem recently. However, the success of them implicitly funds on the assumption that the information of domains are fully transferrable. If the assumption is not satisfied, the effect of negative transfer may degrade domain adaptation. In this paper, a better learning network has been proposed by considering three tasks - domain adaptation, disentangled representation, and style transfer simultaneously. Firstly, the learned features are disentangled into common parts and specific parts. The common parts represent the transferrable features, which are suitable for domain adaptation with less negative transfer. Conversely, the specific parts characterize the unique style of each individual domain. Based on this, the new concept of feature exchange across domains, which can not only enhance the transferability of common features but also be useful for image style transfer, is introduced. These designs allow us to introduce five types of training objectives to realize the three challenging tasks at the same time. The experimental results show that our architecture can be adaptive well to full transfer learning and partial transfer learning upon a well-learned disentangled representation. Besides, the trained network also demonstrates high potential to generate style-transferred images.






§ INTRODUCTION



To decrease the demand on relabeling extra training data when applying the trained network in a new domain, some transfer learning methods for domain adaptation have been proposed in the past decade. Most of the methods aim at reducing the domain shift or minimizing the difference among domain feature distributions. The key idea is to learn deep feature transformations to map inputs from different domains into a common feature space so that the extracted features are both domain-invariant and class-discriminative. Some metrics of domain shift have been used in these methods such as maximum mean discrepancy (MMD) <cit.>, multi kernel variant of MMD (MK-MMD) <cit.>, and correlation distances <cit.>. Recently, along with the success and understanding of the Generative Adversarial Network (GAN) <cit.> for many generative tasks, researchers have turned to borrow the idea of adversarial learning to perform domain adaptation. Based on adversarial learning, domain adaptation problem is modeled as a minimax game between a domain discriminator and a feature extractor. The feature extractor is trained to extract features which can not only minimize the classification loss but also fool the domain discriminator.

Most aforementioned methods focus on learning the common feature representation to bridge and transfer learnable knowledge from a source domain to a target domain <cit.>. The success of these methods implicitly funds on the assumption that all information from a source domain are fully transferable. However, some domain specific features are only suitable to characterize the domain properties. Negative effects may happen and degrade domain adaptation if the domain specific features are transferred. Moreover, partial transfer learning <cit.> is another new concern where the target label space is only a subset of the source label space. Due to the effects of outlier source classes, simplify matching the whole source and target domains as the previous methods would also result in negative transfer. In these cases, learned representation should preferably be disentangled into the transferable and non-transferable features. However, very few studies mention it while solving domain adaptation problems.



In this paper, we combine the disentanglement task with domain adaptation, and also introduce the concept of feature exchange across domains that is useful for style-transfer (or image-to-image translation). The designed idea of our framework is illustrated in Fig. <ref> where three tasks: disentangled representation learning, domain adaptation and style transfer are embbed into a unified framework. Firstly, to relieve the negative transfer problem, we aim to disentangle the domain specific features from the common features which are transferable across the domains. To do so, the designed deep framework divide the representation into two parts: common and specific parts. The common features show the shared contents between two domains. Thus, the domain feature distributions of the common parts are expected to be matched after domain adaption. The specific features mainly reveal the domain properties such as style and appearance. By transferring messages through the common parts rather than the specific parts, we could achieve better transfer learning with less negative propagation. Furthermore, instead of directly ignoring the specific features, we then proposed to integrate style transfer into our domain adaptation framework. As shown in Fig. <ref>, by exchanging features across domains, the style-transferred images can be synthesized from specific features and exchanged common features. In this paper, we also show how these synthesized images can improve domain adaptation. To sum up, our contributions can be summarized as follows:


  * 3-in-one framework: As far as our knowledge, this is the first work discussing about the relationship of three tasks: disentangled representation learning, domain adaptation and style transfer and combining them into a unified framework.

  * Disentangled Representation Learning for Domain Adaptation: By disentangling the features, we aim to decompose the feature space into common parts which are suitable for domain transfer and specific parts which are not transferable. Therefore, we can reduce the influence of negative transfer.

  * Feature Exchange for Style Transfer: We propose a new concept of feature/style exchange across domains. In our framework, the specific part of one domain can be combined with the common part of the another one to form a new feature representation that later on can be used to generate a style-transferred image through a GAN-based framework. With this idea, our framework can not only enhance the transferability of common features but also be extended to solve multi-domain image-to-image translation problems.

  * Feed-back design: Based on the ability to generate a style-transferred image, we propose a novel feed-back design and a semantic consistency loss to enhance the transferability of the learned common parts and preserve the semantic information during feature exchange process.


In brief, the paper organization is as follows: Section <ref> summarizes the related works. Section 3 presents the proposed method. Section 4 presents the datasets and the experimental results. Finally, we conclude this work in Section 5.



§ RELATED WORKS


Domain Adaptation: A large number of domain adaptation methods have been proposed over the recent years. We divide them into two main groups: 1) finding the mapping between source and target domains, and 2) finding the shared latent space between two domains. For the first group, some methods focus on learning the mapping from a source domain to a target domain in the feature level <cit.> or the image level <cit.>. Some other methods choose to learn the inverse mapping from target to source <cit.>. Recently, to make the adaptation system more robust and general, the authors in <cit.> proposed to combine both mapping directions in a unified architecture. For the second group, researchers focus on minimizing the distance between two domains in feature space by using first order statistic <cit.>, or higher order statistic <cit.>. More recently, domain shift between two domains could be further reduced by applying adversarial learning. The main concept is to find the shared latent space so that a strong domain classifier could not distinguish source samples and target samples. There are variants of training strategies for adversarial learning. In <cit.>, the authors proposed to combine a domain confusion loss and softmax cross-entropy loss to train the network so that the network could transfer knowledge not only across domains but also tasks. However, the method still needs sparsely labeled data in target domain. In <cit.>, the authors proposed an unsupervised adversarial discriminative domain adaptation framework. According to their result, an adversarial learning framework based on asymmetric feature mappings for source and target can outperform the one based on a symmetric mapping. To realize adversarial learning, the choice of adversarial loss function is another issue. In <cit.>, the authors directly treat the domain discriminator loss as the adversarial loss to learn the optimal feature mapping; while the authors in <cit.> train the optimal feature mapping with the standard discriminator loss function with inverted labels.
 
Disentangled Representation Learning: Recently, the research group in <cit.> argued that most of the conventional domain adaptation methods learn the common representations of source and target domains without considering the negative influence from the domain specific characteristics. If the network transfers the negative effects, we may not be able to learn a well generalized common feature representation. Therefore, the authors proposed a two-stage neural network learning algorithm to learn a multi-part hidden layer where individual parts can be disentangled or combined for different tasks in different domains. Also aiming at answering the fundamental question “what to transfer”, Domain Separation Network <cit.>, proposed to integrate a private network to learn the private subspace for each domain and a shared network to extract the shared representation subspace across domains. To force the private subspace and shared subspace to be independent, a difference loss is introduced. After domain separation, the standard adversarial loss and classification loss can be applied only in the shared subspace.

Partial Transfer Learning: Partial transfer learning was proposed in <cit.>, where the target domain label space is a subspace of the source domain label space. Because the extra source classes might cause negative transfer when classifying the target domain, it makes the domain adaptation problem more challenging. In this work, in order to solve the partial transfer problem, instead of using single-discriminator domain adversarial network, the authors proposed to use multi-discriminator domain adversarial network, each discriminator is responsible for matching the source and target domain data associated with each label.





§ THE PROPOSED DOMAIN ADAPTATION NETWORK


In this paper, we firstly focus on the problem of unsupervised domain adaptation. Particularly, we aim to train a target domain classifier F_C that can correctly predict the label for the target data X_T by leveraging the source data X_S and source labels Y_S. Unlike the conventional setting of domain adaptation, we pay further attention on (a) transfer learning of common feature components and (b) partial transfer learning. For case (a), we argue that disentangled representation learning is necessary to decompose the common feature parts and specific parts of domains. For case (b), we design our framework to match the cluster-based distributions between source and target domains instead of matching the whole domain distribution. Finally, we proposed to exchange the specific feature parts across domains and synthesize style-transferred images. These style-transferred images are further used to improve domain adaptation. Our framework is illustrated in Fig. <ref>. Two key function modules in our feature extraction and image generation framework are summarized as follows:


  * Disentangled Representation Learning: The feature extraction networks, F_S and F_T, have the ability to disentangle the feature maps into different semantic parts including the common parts (C_S and C_T) and specific parts (S_S and S_T). We denote the decomposition processes as C_S=F_S^C(x_S), C_T=F_T^C(x_T), S_S=F_S^S(x_S), and S_T=F_T^S(x_T). As an instance, in our digit classification experiments, the common parts imply the semantic features of “digits”; the specific parts mainly reveal the domain style such as the writing style and appearance. Each part could be utilized in a different way to connect two domains and finally helps to transfer knowledge for domain adaptation.

  * Feature exchange across domains: We combine the specific part of one domain with the common part of the other one to synthesize a new feature representation as shown in Fig. <ref>. Next, based on a learnable GAN-based network, our system can generate a style-transferred image given a synthesized feature representation. The style-transferred image, keeping the original image content but different image style, would play important roles to bridge domains. This process also enable our system to transfer the labels from source to target.


Besides feature extractors and generators, our framework also consists of three discriminators and one standard classifier as illustrated in Figure <ref>. Based on the supervised labels in the source domain, the standard classifier ( F_C) and the source common features should be well trained in order to correctly classify the source domain samples. In addition, three adversarial discriminators  D_S, D_T and  D_F are introduced for representation adaptation in both the image level and feature level.  D_S aims to distinguish between real source images X_S and style-transferred target image, denoted as {G_S(S_S,C_T)}. In the similar way, D_T aims to discriminate between X_T and {G_T(S_T,C_S)}. By applying adversarial learning in the image level with D_S and D_T, we hope that the style-transferred images should not be distinguished from the real ones. This property is quite helpful for domain adaptation, which would be explained later. Moreover, we also apply adversarial learning in the feature level by D_F, which is designed to distinguish between the common parts C_S and C_T of source and target domains. With the helps of the adversarial discriminators D_F, we can match the feature distribution of the source common part C_S and the target common part C_T. To train our domain adaptation network, we designed and implemented 5 types of objective losses as described in the following subsections.





 §.§ Adversarial Losses


Our domain adaptation framework works like the conditional GANs <cit.> in which the conditional variables are common parts C and specific parts S. However, we do not use any random noise as GAN inputs. All the features come from data themselves. For domain adaptation, we apply adversarial losses <cit.> and design three adversarial discriminators (D_S, D_T, D_F) to jointly match the distributions of two domains in both the image and feature levels. However, our discriminators not only aim to distinguish between source and target domains but also try to classify the input data. Similar to the design in <cit.>, our adversarial discriminators are (N_C+1)-way classifiers with N_C binary nodes to indicate N_C content classes and an extra class for type discrimination. For D_S and D_T, the extra class represents a real (1) or synthesized (0) image; For D_F, it indicates the source (1) or target (0) domain. With this design, the discriminators could be more powerful and help our system to match the cluster-based distribution between domains instead of matching the whole domain distribution as the traditional discriminator <cit.>. Note that, due to image style transfer, our network can still train D_T even though we do not have supervised labels in the target domain. Indeed, the label information is borrowed from the annotation in the source domain.
Accordingly, we have defined three adversarial losses for training: (a) the feature level loss L_(adv_fea), (b) the image-level loss in the source domain L_(adv_img)^S, and (c) the image-level loss in the target domain L_(adv_img)^T.



  §.§.§ Feature Level Loss (L_adv_fea)

Since our discriminator plays two roles at the same time, a type classifier and a content classifier, L_adv_fea is composed of a type loss L_type_D_F and a classification loss L_cls_D_F as defined in equation <ref>.

    L_adv_fea(F_S, F_T, D_F)= L_type_D_F(F_S, F_T, D_F) + L_cls_D_F(F_S, D_F).

In (<ref>), the type loss L_type_D_F is optimized by the following function:

    min_F_S, F_Tmax_D_F E_x_Slog(D_F^N_C+1(F_S^C(x_S))) + E_x_T log(1-D_F^N_C+1(F_T^C (x_T))).

Here, D_F^(N_C+1)(.), the (N_C+1)^th output of discriminator D_F, works as a domain classifier and predicts the domain label (1 for source and 0 for target) for each input sample. F_S^C (x_S ) and F_T^C (x_T) are the common feature parts of a source sample x_S and a target sample x_T. On the other hand, the classification loss L_cls_D_F is defined as a standard softmax cross-entropy. That is,

    L_cls_D_F(F_S, D_F)=min_F_S, D_F -E_x_S∑_i=0^N_Cy_S^i.log(σ(D_F^i(F_S^C(x_S)))).

where D_F^i(.), the i^th output of discriminator D_F, works as a content classifier and predicts the probability of the i^th class. y_S^i is the supervised label for the input sample x_S. In (<ref>), σ(.) denotes the softmax function.



  §.§.§ Source Domain Image-level Loss (L_adv_img^S)

Like L_adv_fea, L_adv_img^S is also composed of a type loss L_type_D_S and a classification loss L_cls_D_S. That is

    L_adv_img^S(F_S, F_T, G_S, D_S)= L_type_D_S(F_S, F_T, G_S, D_S) + L_cls_D_S(D_S),


The type loss L_type_D_S related to discriminator D_S is also defined by traditional adversarial loss function. That is

    min_F_S, F_T, G_Smax_D_SE_x_Slog(D_S^N_C+1(x_S)) + E_S_S,C_Tlog(1-D_S^N_C+1(G_S(S_S,C_T))).

In (<ref>), D_S^(N_C+1)(.) is a type classifier used to discriminate a real or synthesized image. G_S (S_S, C_T) generates an image by concatenating a source specific feature S_S and a target common feature C_T. Note that S_S=F_S^S(x_S) and C_T=F_T^C (x_T).

Similar to (<ref>), L_cls_D_S(D_S), is defined as in equation (<ref>).

    L_cls_D_S(D_S)=min_D_S -E_x_S∑_i=0^N_Cy_S^i.log(σ(D_S^i(x_S))).




  §.§.§ Target Domain Image-level Loss (L_adv_img^T)

Target adversarial loss L_adv_img^T shares the same design concept as L_adv_img^S. It is defined as

    L_adv_img^T(F_S, F_T, G_T, D_T)= L_type_D_T(F_S, F_T, G_T, D_T) + L_cls_D_T(F_S, F_T, G_T, D_T),

where the type loss L_type_D_T and image content classification loss L_cls_D_T are

    L_type_D_T(F_S, F_T, G_T, D_T)=
    
    E_x_Tlog(D_T^N_C+1(x_T)) + E_S_T,C_Slog(1-D_T^N_C+1(G_T(S_T,C_S))).


    L_cls_D_T(F_S, F_T, G_T, D_T)= -E_S_T,C_S∑_i=0^N_Cy_S^i.log(σ(D_T^i(G_T(S_T,C_S)))).

Here, it is worth mentioning that the supervised class label y_S^i for training the target discriminator D_T is borrowed from the source domain. In (<ref>), we generate many target-style images with source labels so that D_T could be trained.



 §.§ Feedback Losses and Reconstruction Losses


Because our framework is designed in the form of a convolutional auto-encoder, in order to make sure the learned features are generalized enough, we also proposed to minimize the reconstruction losses for both domains as shown in (<ref>).

    min_G_S,F_S,G_T,F_TE_x_S||x_S-G_S(F_S(x_S))||_2^2+E_x_T||x_T-G_T(F_T(x_T))||_2^2.


Besides, inspired by the cycle-consistency loss <cit.>, we proposed feedback losses to enforce the learning of feature extractors and image generators. In detail, we input a combined feature map (S_S, C_T) into a generator G_S to generate a source-style image. Ideally, if we input the synthesized image into feature extractor F_S and get its feature map S_S', C_T', we would hope the two feature maps S_S', C_T' and S_S, C_T are consistent. The similar concept could also be applied to the target domain. These feedback constraints could be integrated in our learning step by imposing an L2 penalty term according to the feedback errors as in (<ref>) and (<ref>). 

    L_feedback^S(F_S, F_T, G_S)= 
    
    E_S_S, C_T(||S_S-F_S^S(G_S(S_S,C_T))||_2^2+||C_T-F_S^C(G_S(S_S,C_T))||_2^2).


    L_feedback^T(F_S, F_T, G_T)= 
    
    E_S_T, C_S(||S_T-F_T^S(G_T(S_T,C_S))||_2^2+||C_S-F_T^C(G_T(S_T,C_S))||_2^2).




 §.§ Semantic consistency loss

Our framework allows the generated target images to inherit the source labels and enables the transfer of sample labels. Furthermore, based on the supervised label transfer, discriminators D_S and D_T could be trained. By leveraging D_S and D_T, we proposed a new semantic consistency loss to improve domain adaptation. If our network is well trained, we expect the classification result D_S(G_S(S_S, C_T|_X_T) of the generated style-transferred image G_S(S_S, C_T|_X_T) should be consistent with D_T(X_T), where C_T|_X_T is the common feature vector extracted from sample X_T. Therefore, in order to encourage this kind of semantic consistency, we introduce the semantic consistency loss as follows:

    L_Sem(F_S, F_T, G_S, D_S, D_T)= E_S_S,X_T||D_T^1→ N_C(X_T)-D_S^1→ N_C(G_S(S_S, C_T|_X_T))||_2^2.


Semantic consistency loss plays the important role to connect both domains. To reduce the semantic consistency loss, our feature extraction networks F are forced to well disentangle the Common and Specific feature components. Meanwhile, the generators G are forced to perform style transfer well.



 §.§ Entropy minimization losses

The classification ability of the discriminators D_S, D_T and D_F is a critical point in our network. However, so far, we only base on the source sample annotation for training. The result might be acceptable for D_S and D_F but might not be perfect for D_T due to the lack of true labels. Thus, to enhance the classification ability, we look for the help from unsupervised methods <cit.> and integrate the concept of entropy minimization <cit.> into our network training.

If we treat the output vector of D_*^1→ N_C∈{D_S^1→ N_C, D_T^1→ N_C, D_F^1→ N_C} as a probability distribution, its entropy can be measured by

    H(σ(D_*^1→ N_C(x_*^i)))= -∑_j=1^N_Cσ(D_*^j(x_*^i)).log(σ(D_*^j(x_*^i))),

where N_C is the class number and σ(.) denotes the softmax function. For an input sample x_*^i, if the corresponding entropy is small, it implicitly means the sample is well classified from an unsupervised viewpoint. Thus, we might enhance the classification ability by minimize the summarization of the entropies of many samples. To utilize this property in our training, we define the three entropy loss terms corresponding to three discriminators D_S, D_T, and D_F as follows:

    L_Entropy(F_S, F_T, G_S, D_S, D_T, D_F)= E_S_S, C_TH(σ(D_S^1→ N_C(G_S(s_S^i,c_T^i))) 
    
    +E_x_TH(σ(D_T^1→ N_C(x_T^i))+E_C_TH(σ(D_F^1→ N_C(c_T^i)).


By minimizing the entropy penalty in (<ref>), our system has two achievements. (1) We can train the three discriminators, the classification part, by unlabeled target samples. (2) The feature extractors are trained to from a well-clustered feature distribution over many classes. These properties improve domain adaptation especially in the partial transfer case. To understand the importance of entropy minimization, we visualize the learned features when training with and without L_Entropy in Fig. <ref> and Fig. <ref>. We may find the margins among the clusters are clear and well separated when the effect of L_Entropy is considered in the training process.



 §.§ Classification loss

The last loss we apply in our learning is the standard classification loss. It uses the labeled source samples to train the classifier F_C in the common feature domain and predict the final output label for a given testing sample. For the N_C-way classification, the multiple-class classification loss are defined as

    L_cls(F_S, F_C)=min_F_S, F_C -E_C_S∑_i=0^N_Cy_S^i.log(σ(F_C^i(C_S))).




§ EXPERIMENTS AND DISCUSSIONS





 §.§ Setup

In order to evaluate the effectiveness of our framework, we validate it by performing domain adaptation on three standard digit datasets: MNIST <cit.>, USPS <cit.>, and SVHN <cit.> which contain 10 classes of digits; and two traffic sign datasets: Syn-Signs <cit.> and GTSRB <cit.> which contain 43 classes of traffic signs. The details of these datasets are described in Supplemental Material A.

With these datasets, we take into account the following unsupervised transfer scenarios. (1) MNIST → USPS:  because USPS and MNIST follow very different distribution, to accelerate experiments, we follow the training protocol created in <cit.>, randomly sampling 1800 images in USPS and 2000 images in MNIST. To reduce the high variance effect in performance of random sampling, we run each experiment five times and report the average performance. (2) SVHN → MNIST: we use the full training sets. All images were rescaled into 32x32 and pixels were normalized to [0, 1] values, and only the labels from source are available during training. (3) Syn-Signs → GTSRB: we also use full training sets. The region of interest around the sign is extracted and rescaled into 40x40 from each image in GTSRB to match those of Syn-Signs.

Two transfer learning problems are focused in our experiments are full transfer learning and partial transfer learning where the target label space is a subset of source label space. For the case of partial transfer learning, we randomly select 5 classes to form the target domain data, and only the scenario MNIST → USPS is considered.

Architecture. For all of these experiments, we simply modify LeNet architecture provided in Caffe source code <cit.> as our extractors, and use the same structure as DCGAN <cit.> for the generators. The details of network architectures are given in Supplemental Material B, the training procedures and hyperparameters are discussed in Supplemental Material C.









 §.§ Full transfer learning

The results of our experiment are provided in Table <ref>, we observe that our method performs well in all scenarios. It is worth to note that, in order to have a fair comparison, we only compare to the methods which have the simple network architecture similar to ours.

Feature visualization. To demonstrate the distributions of our learned features, we use t-SNE <cit.> projection. As shown in Fig. <ref>, the common features from target and source domains are matched together and grouped into 10 main clusters clearly, especially for the simple case MNIST → USPS. This means that our framework not only can learn the common features which cannot be distinguished between domains but also have ability to match the cluster-based distributions of two domains.  On the contrary, the specific features from target and source domains are separated well and far away from common features. It means that our framework can learn the domain specific characteristics and these characteristics are different from the common features even we did not apply any difference loss such as orthogonality constraint on these features.

With and without semantic consistency loss. In Fig. <ref>, we show the comparison of the style-transferred images produced by the generators G_S and G_T with and without semantic consistency loss. Without semantic consistency, the style-transferred images (Fig. <ref> and <ref>) seem to have the same style as the real ones, but they fail at preserving the semantic information. They might deliver ambiguous information to the classifier and discriminators. As shown in (Fig. <ref> and <ref>), with semantic consistency loss, the style-transferred images successfully both preserve the semantic information and depict the style information.




Visualizing the common and specific parts in image domain. In Fig. <ref> we show some examples for SVHN → MNIST case. We also try to visualize the common and specific parts in image domain by inhibiting the remaining part before inputting to the Generators. For example, if we want to visualize the common parts, we will set all specific parts equal to zero then concatenate them and input into the corresponding Generator. As shown in the Fig. <ref> and <ref>, the common parts of each domain only store the information of “digits”. Meanwhile, the specific parts encode the style information such as the contrast, color, size, ... as shown clearly in Fig. <ref>. For the target's specific parts shown in Fig. <ref>, all the images are almost same because the style of all the images in target domain (MNIST) are same. This experiment partly demonstrates our aforementioned statements about our framework's abilities to learn disentangled representations and transfer the style across domain.

The visualizing results for Syn-Signs to GTSRB are given in the Supplemental Material D.











 §.§ Partial Transfer Learning

In order to quickly test whether our framework is capable of solving partial transfer learning problem, we still use the digit datasets and randomly select 5 classes to form the target domain data (USPS), and apply our framework with 3 different settings: (1) without feedback and entropy minimization losses, (2) without entropy minimization losses and (3) proposed method. Based on the results in Table <ref> and the t-SNE visualization in Fig. <ref>, we can understand the importance of L_feedback and L_Entropy in our framework especially for partial transfer learning case. As aforementioned, by enhancing the harmony and the stability of extractors and generators, L_feedback improves the performance significantly by 14.28%. Without L_feedback, the outlier source clusters seem to cause certain difficulties to transferring process, as shown in Fig. <ref>, the corresponding common clusters between source and target are not aligned well and some clusters are mismatched. With L_feedback, as shown in Fig. <ref>, the corresponding classes between source and target are matched, however it still has ambiguous areas between classes. In Fig. <ref>, we can recognize that L_Entropy solves this problem and boosts the performance (5.28%) by clustering semantic classes together and thereby increasing the distance between clusters and then reducing the ambiguity. Moreover, comparing the performance and the feature distribution of our method for two transferring cases (full transferring and partial transferring), we can partly assert that our network can solve the partial transferring problem efficiently.











§ CONCLUSIONS


In this paper, we proposed a unified framework for unsupervised domain adaptation that can disentangle the feature domain into two main parts: common features across domains, and specific features of each domain. The common features are used to embed the content information that are useful for classification across domains, and the specific features are used to encode the domain specific characteristics. To enhance the transferability of common features, we proposed the novel idea for feature exchange across domains which also allows us to embed style transfer function into our framework. Besides, we also introduced the feedback design and semantic consistency loss to improve the harmony and the stability of the components in the framework. Last but not least, entropy minimization losses were applied in our framework as a refining method to encourage the self-clustering ability in the target domain and thereby make the transfer process easier. The experiments confirm the improvement of our network in solving both of full transfer problem and partial transfer problem. Besides, the results also show the potential of our network in image style transfer.


splncs






§ SUPPLEMENTARY MATERIAL



 §.§ Datasets



The datasets used in this paper are described in Table  <ref>. It should be noted that for GTSRB dataset, the original number of training data is 39,209 and testing data is 12,630. However, the label for testing data is not available, we randomly select 31,367 samples from original training set to form the new training set and the rest for evaluation. In addition to the datasets listed in Table  <ref>, there are other datasets for domain adaptation such as OFFICE and Office+Caltech. Due to high image variations and small amounts of data, some works pre-train their models via other big datasets before performing domain adaptation in these datasets. However, by using other pre-trained models, it may not reveal pure message transfer between two domains. Thus, we insist to train our network from scratch. For fair comparison, these datasets are not included in our experiment now.






 §.§ Network Architectures


Our network architectures are shown in <Ref>.





































 §.§ Training Procedures and Hyperparameters


Because we would like to train our network from scratch, before training the whole network, we train each part of it one by one by the following steps:


  * Step 1: Train a good classifier and feature extraction using source data with the learning rate 0.01. (20,000 iteration for MNIST to USPS, 43,000 iteration for SVHN to MNIST, and 50,000 iteration for Syn-Signs to GTSRB)

  * Step 2: Add the generator into the framework and continue train it using reconstruction loss (for both source and target data) and classification loss (for source data only) with the learning rate 0.01. (5,000 iteration for MNIST to USPS, 16,000 iteration for SVHN to MNIST, and 7,000 iteration for Syn-Signs to GTSRB)

  * Step 3: Duplicate the network (one for source and one for target) and add the feature discriminator into the framework, continue train it using L_adv_fea (loss_weight: 1), reconstruction loss (loss_weight: 1) and classification loss (loss_weight: 10) with the learning rate 0.001. (25,000 iteration for MNIST to USPS, 25,000 iteration for SVHN to MNIST, and 27,000 iteration for Syn-Signs to GTSRB)

  * Step 4: Add the image discriminators into the framework and initilize these discriminators using trained generator's weights, continue train the framework using L_adv_img (loss_weight: 1), L_adv_fea (loss_weight: 1), reconstruction loss (loss_weight: 1) and classification loss (loss_weight: 10) with the learning rate 0.001. (5,000 iteration for MNIST to USPS, 7,000 iteration for SVHN to MNIST, and 9,000 iteration for Syn-Signs to GTSRB)


When we achieve the initial weights for the whole parts of our network by the previous process, we start to train it end-to-end by the following steps:


  * Step 1: We first warm-up the whole network by continue training it without L_Entropy with the learning rate 0.001. (39,000 iterations for MNIST to USPS, 129,500 iteration for SVHN to MNIST, and 36,000 iteration for Syn-Signs to GTSRB)

  * Step 2: We copy (replace) target network’s weights by the trained source network’s weights and then re-train without L_Entropy with the learning rate 0.001. (20,000 iterations for MNIST to USPS, 39,600 iteration for SVHN to MNIST, and 25,600 iteration for Syn-Signs to GTSRB)

  * Step 3: We refine the network by training with L_Entropy loss and learning rate 0.0001. (2,000 iterations for MNIST to USPS, 16,000 iteration for SVHN to MNIST, and 6,000 iteration for Syn-Signs to GTSRB)


During training the whole network, we can repeat step 2 and step 3 to achieve better initialization. As the network converges, we can also reduce the learning rate by ten times and continue training to get higher performance.



 §.§ Visualizing Results for Syn-Signs → GTSRB



In Fig. <ref> we show some examples for Syn-Signs → GTSRB case. Here, we also visualize the common and specific parts in image domain by inhibiting the remaining part before inputting to the Generators. As shown in the Fig. <ref> and <ref>, the common parts of each domain store the information of “traffic signs”. Meanwhile, the specific parts encode the style information such as the background, shape, ... as shown in Fig. <ref> and  <ref>.


