Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Unsupervised domain adaptation aims to generalize the hypothesis trained in a source domain to an unlabeled target domain. One popular approach to this problem is to learn domain-invariant embeddings for both domains. In this work, we study, theoretically and empirically, the effect of the embedding complexity on generalization to the target domain. In particular, this complexity affects an upper bound on the target risk; this is reflected in experiments, too. Next, we specify our theoretical framework to multilayer neural networks. As a , we develop a strategy that mitigates sensitivity to the embedding complexity, and empirically achieves performance on par with or better than the best layer-dependent complexity tradeoff. Domain adaptation is critical in many applications where collecting large-scale supervised data is prohibitively expensive or intractable, or where conditions at prediction time can change. For instance, self-driving cars must be robust to different weather, change of landscape and traffic. In such cases, the model learned from limited source data should ideally generalize to different target domains. Specifically, unsupervised domain adaptation aims to transfer knowledge learned from a labeled source domain to similar but completely unlabeled target domains. One popular approach to unsupervised domain adaptation is to learn domain-invariant representations (; ;), by minimizing a divergence between the representations of source and target domains. The prediction function is learned on these "aligned" representations with the aim of making it domain-independent. A series of theoretical works justifies this idea (; ; ;). Despite the empirical success of domain-invariant representations, exactly matching the representations of source and target distribution can sometimes fail to achieve domain adaptation. For example, show that exact matching may increase target error if label distributions are different between source and target domain, and propose a new divergence metric to overcome this limitation. establish lower and upper bounds on the risk when label distributions between source and target domains differ. point out the information lost in non-invertible embeddings, and propose different generalization bounds based on the overlap of the supports of source and target distribution. In contrast to previous analyses that focus on changes in the label distributions or joint support, we study the effect of embedding complexity. In particular, we show a general bound on the target risk that reflects a tradeoff between embedding complexity and the divergence of source and target domains. A too powerful class of embeddings can in overfitting the source data and the matching of source and target distributions, ing in arbitrarily high target risk. Hence, a restriction is needed. We observe that indeed, without appropriately constraining the embedding complexity, the performance of state-of-the-art methods such as domain-adversarial neural networks can deteriorate significantly. Next, we tailor the bound to multilayer neural networks. In a realistic scenario, one may have a total depth budget and divide the network into an encoder (embedding) and predictor by aligning the representations of source and target in a chosen layer, which defines the division. In this case, a more complex encoder necessarily implies a weaker predictor, and vice versa. This tradeoff is reflected in the bound and, we see that, in practice, there is an "optimal" division. To better optimize the tradeoff between encoder and predictor without having to tune the division, we propose to optimize the tradeoffs in all layers jointly via a simple yet effective objective that can easily be combined with most current approaches for learning domain-invariant representations. Implicitly, this objective restricts the more powerful deeper encoders by encouraging a simultaneous alignment across layers. In practice, the ing algorithm achieves performance on par with or better than standard domain-invariant representations, without tuning of the division. Empirically, we examine our theory and learning algorithms on sentiment analysis (Amazon review dataset), digit classification (MNIST, MNIST-M, SVHN) and general object classification (Office-31). In short, this work makes the following contributions: • General upper bounds on target error that capture the effect of embedding complexity when learning domain-invariant representations; • Fine-grained analysis for multilayer neural networks, and a new objective with implicit regularization that stabilizes and improves performance; • Empirical validation of the analyzed tradeoffs and proposed algorithm on several datasets. For simplicity of exposition, we consider binary classification with input space X ⊆ R n and output space Y = {0, 1}. Define H to be the hypothesis class from X to Y. The learning algorithm obtains two datasets: labeled source data X S from distribution p S, and unlabeled target data X T from distribution p T. We will use p S and p T to denote the joint distribution on data and labels X, Y and the marginals, i.e., p S (X) and p S (Y). Unsupervised domain adaptation seeks a hypothesis h ∈ H that minimizes the risk in the target domain measured by a loss function (here, zero-one loss): We will not assume common support in source and target domain, in line with standard benchmarks for domain adaptation such as adapting from MNIST to MNIST-M. A common approach to domain adaptation is to learn a joint embedding of source and target data . The idea is that aligning source and target distributions in a latent space Z in a domain-invariant representations, and hence a subsequent classifier f from the embedding to Y will generalize from source to target. Formally, this in the following objective function on the hypothesis h = f g:= f • g, where G is the class of embedding functions g to Z, and we minimize a divergence d between the distributions p ) of source and target after mapping to Z: The divergence d could be, e.g., the Jensen-Shannon or Wasserstein distance . Ben - introduced the H∆H-divergence to bound the worst-case loss from extrapolating between domains. Let R D (h, h) = E x∼D [(h(x), h (x))] be the expected disagreement between two hypotheses. The H∆H-divergence measures whether there is any pair of hypotheses whose disagreement (risk) differs a lot between source and target distribution. Definition 1. (H∆H-divergence) Given two domain distributions p S and p T over X, and a hypothesis class H, the H∆H-divergence between p S and p T is The H∆H-divergence is determined by the discrepancy between source and target distribution and the complexity ofthe hypothesis class H. For a hypothesis class H: X → {0, 1}, the disagreement between two hypotheses is equivalent to the exclusive or function. Hence, one can interpret the H∆H-divergence as finding a classifier in function space H∆H = H ⊕ H which attempts to maximally separate one domain from the other . A restrictive hypothesis space may in small H∆H-divergence even if the source and target domain do not share common support. This divergence allows us to bound the risk on the target domain: Theorem 2. For all hypotheses h ∈ H, the target risk is bounded as where λ H is the best joint risk Similar exist for continuous labels . Theorem 2 is an influential theoretical in unsupervised domain adaptation, and motivated work on domain invariant representations. For example, recent work applied Theorem 2 to the hypothesis space F that maps the representation space Z induced by an encoder g to the output space: where λ F (g) is the best hypothesis risk with fixed g, i.e., λ The F∆F divergence implicitly depends on the fixed g and can be small if g provides a suitable representation. However, if g induces a wrong alignment, then the best hypothesis risk λ F (g) is large with any function class F. The following example will illustrate such a situation, motivating to explicitly take the class of embeddings into account when bounding the target risk. We begin with an illustrative toy example. Figure 1 shows a binary classification problem in 2D with disjoint support and a slight shift in the label distributions from source to target: p S (y = 1) = p T (y = 1) + 2. Assume the representation space Z is one dimensional, so the embedding g is a function from 2D to 1D. If we allow arbitrary, nonlinear embeddings, then, for instance, the embedding in Figure 1(b), together with an optimal predictor, achieves zero source loss and a zero divergence which is optimal according to the objective in equation. But the target risk of this combination of embedding and predictor is maximal: If we restrict the class G of embeddings to linear maps g(x) = Wx where W ∈ IR 1×2, then the embeddings that are optimal with respect to the objective are of the form W = [a, 0]. Together with an optimal source classifier f, they achieve a non-zero value of 2 for objective due to the shift in class distributions. However, these embeddings retain label correspondences and can thus minimize target risk. This example illustrates that a too rich class of embeddings can "overfit" the alignment, and hence lead to arbitrarily bad solutions. Hence, the complexity of the encoder class plays an important role in learning domain invariant representations. Motivated by the above example, we next expose how the bound on the target risk depends on the complexity of the embedding class. To do so, we apply Theorem 2 to the hypothesis h = f g: This bound differs in two ways from the previous bound (equation), which was based only on F: the best in-class joint risk now minimizes over both F and G, i.e., which is smaller than λ F (g) and reflects the fact that we are learning both f and g. In return, the divergence term d F G∆F G (p S, p T) becomes larger than the one in equation. To better understand these tradeoffs, we will reformulate bound to be more interpretable. To this end, we define a version of the H∆H-divergence that explicitly measures variation of the embeddings in G: For two domain distributions p S and p T over X, an encoder class G, and predictor class F, the F G∆G -divergence between p S and p T is Importantly, the F G∆G -divergence is smaller than the FG∆FG-divergence, since the two hypotheses in the supremum, f g and f g, share the same predictor f. Theorem 4. For all f ∈ F and g ∈ G, where λ F G (g) is the best in-class joint risk defined as We prove all theoretical in the Appendix. This target generalization bound is small if (C1) the source risk is small, (C2) the latent divergence is small (because the domains are well-aligned and/or F is restricted), (C3) the complexity of G is restricted to avoid overfitting of alignments, and (C4) good source and target risk is in general achievable with F and G. Comparison to Previous Bounds. The last two terms in Theorem 2 express a similar complexity tradeoff, but with respect to the overall hypothesis class H, which here combines encoder and predictor. Directly applying Theorem 2 to the composition H = FG (equation) treats both jointly and does not make the role of the embedding as explicit as Theorem 4. The recent bound assumes a fixed embedding g and focuses on the predictor class F. As a , it captures embedding complexity even less explicitly: the first two terms in bound and Theorem 4 are the same. The last term in, λ F (g), contains the target risk with the given g. Hence, bound replaces (C3) and (C4) above by saying F and the specific g (which is much harder to control since in practice it is also optimized) can achieve good source and target risk. In contrast, Theorem 4 states an explicit complexity penalty on the variability of the embeddings, and uses the fixed g only in the source risk, which can be better estimated empirically. If F is not too rich, the latent divergence can be empirically minimized by finding a well-aligned embedding. Hence, we can minimize the upper bound in Theorem 4 by minimizing the usual source loss and domain-invariant loss and by choosing F and G appropriately to tradeoff the complexity penalty d F G∆G, the latent divergence (which increases with complexity of F and decreases with complexity of G), and the best in-class joint risk (which decreases with complexity of F and G). To empirically verify the embedding complexity tradeoff, we keep the predictor class F fixed, vary the embedding class G, and minimize the source loss and alignment objective. Concretely, we train domain adversarial neural networks (DANNs) on the Amazon reviews dataset (Book → Kitchen). Our hypothesis class is a multi-layer ReLU network, and the divergence is minimized against a discriminator. For more experimental details and , please refer to section 6. We train different models by varying the number of layers in the encoder while fixing the predictor to 4 layers. Figure 3.2(a) shows that, when increasing the number of layers in the encoder, the target error decreases initially and then increases as more layers are added. This supports our theory: the smaller encoders are not rich enough to allow for good alignments and λ F G (g), but overly expressive encoders may overfit. Predictor Complexity. Theoretically, the complexity of the predictor class F also affects the generalization bound in Theorem 4. Empirically, we found that the predictor complexity has much weaker influence on the target risk (see experiments in Appendix B). Indeed, theoretically, while the complexity of F affects the latent divergence, if the alignment via g is very good, this divergence can still be small. In addition, the F G∆G -divergence is more sensitive to the embedding complexity than the predictor complexity. This offers a possible explanation for our observations. In the remainder of this paper, we focus on the role of the embedding. Discussion. The in this section indicate that, without constraining the embedding complexity, we may overfit the distribution alignment and thereby destroy label consistency as in Figure 1. The bound suggests to choose the minimal complexity encoder class G that is is still expressive enough to minimize the latent space divergence. Practically, this can be done by regularizing the encoder, e.g., restricting Lipschitz constants or norms of weight matrices. More explicitly, one may limit the number of layers of a neural network, or apply inductive biases via network architectures. For instance, compared to fully connected networks, convolutional neural networks (CNNs) restrict the output representations to be spatially consistent with respect to the input. Due to their wide empirical success, multilayer neural networks have been adopted for learning domain-invariant representations. Next, we adapt the bound in Theorem 4 to multilayer networks. Specifically, we consider the number of layers as an explicit measurement of complexity. This will lead to a simple yet effective algorithm to mitigate the negative effect of very rich encoders. Assume we have an N -layer feedforward neural network h ∈ H. The model h can be decomposed as h = f i g i ∈ F i G i = H for i ∈ {1, 2, . . ., N − 1} where the embedding g i is formed by the first layer to the i-th layer and the predictor f i is formed by the i + 1-th layer to the last layer. We can then rewrite the bound in Theorem 4 in layer-specific form: Latent Divergence in i-th layer This yields N − 1 layer-specific upper bounds. Importantly, minimizing the domain-invariant loss in different layers leads to different tradeoffs between fit and complexity penalties. This is reflected by the following inequalities that relate different layer divisions. Proposition 5. (Monotonicity) In an N -layer feedforward neural network h = f i g i ∈ F i G i = H for i ∈ {1, 2, . . ., N − 1}, the following inequalities hold for all i ≤ j: Proposition 5 states that the latent divergence is monotonically decreasing and the complexity penalty is monotonically increasing with respect to the embedding's depth. This is a tradeoff within the fixed combined hypothesis class H. A deeper embedding allows for better alignments and simultaneously reduces the depth (power) of F; both reduce the latent divergence. At the same time, it incurs a larger F G∆G -divergence. This suggests that there might be an optimal division that minimizes the bound on the target risk. In practice, this translates into the question: in which intermediate layer should we optimize the domain-invariant loss? Figure 3.2(b) shows how the target error changes as a function of the layer division, with a total of n = 8 layers. Indeed, empirically there is an optimal division with minimum target error, suggesting that for a fixed H, i.e., total network depth, not all divisions are equal. If the exact layer-specific bounds could be computed, one could simply select the layer division with the lowest bound. But, this is in general computationally nontrivial. Instead, we take a different perspective. In fact, the layer-specific bounds all hold simultaneously, independent of the layer we selected for distribution alignment. Corollary 6. Let h be an N -layer feedforward neural network h = f i g i ∈ F i G i = H for i ∈ {1, 2, . . ., N − 1}, we have the layer-agnostic bound where λ F G (g) is the best in-class joint risk defined in Theorem 4. The corollary implies that at least one of these bounds should be small. Recall that the bounds depend on how well we can minimize the source risk and align the distributions via a sufficiently powerful embedding, while, at the same time, limiting the complexity of F and G. Corollary 6 points to various algorithmic ideas: Simultaneously optimizing several bounds may in approximately minimizing at least one of them, without having to select an optimal one. We may attain small latent divergence with a deeper encoder, if we achieve to restrict the complexity of G appropriately. It turns out that these two ideas are related. Optimizing the domain-invariant loss with alignment in a specific layer may in large bounds for the other layers, due to the monotonicity of the two divergences (Proposition 5) and potentially non-aligned embeddings in lower layers. Hence, we propose to instead solve a multi-objective optimization problem where we jointly align source and target distributions in multiple layers. Let L ⊆ {1, 2, . . ., N − 1} be a subset of layers. We minimize the weighted sum of divergences, and refer to this objective as Multilayer Divergence Minimization (MDM): This objective encourages alignment throughout the layer-wise embeddings in the network. First, a good alignment minimizes the latent divergence, if F is not too rich. For the lower layers (shallow embeddings), this comes together with a very restricted class of embeddings, and hence limits both latent divergence and complexity penalty. Without the optimization across layers, the embeddings in lower layers are not driven towards alignment. Second, enforcing alignment in lower layers implicitly restricts the deeper embeddings in higher layers, since the embeddings are such that alignment happens early on. This effect may be viewed as an implicit regularization. By this perspective, the bounds for higher layers profit from low latent divergences (deeper embeddings and shallow predictors) and restricted complexity of G. In general, one can simply set L = {1, 2, . . ., N − 1}. To improve computational efficiency, we can sub-sample layers or exclude the first and the last few layers. MDM is simple and general, and can be combined with most algorithms for learning domain-invariant representations. For DANN, for instance, we minimize the divergence in multiple layers by adding discriminators. Existing approaches for learning domain-invariant representations may be distinguised, e.g., by which divergence they measure between source and target domain. Examples include domain adver- sarial learning approaches (; ;), maximum mean discrepancy (MMD) (; 2015; and Wasserstein distance (; ;). Other works improve performance by combining the domain-invariant loss with other objectives. penalize the violation of the cluster assumption. In addition to the shared feature encoder between source and target domain, include private encoders for each domain to capture domain-specific information. propose a domain discriminator that is conditioned on the cross-covariance of domain-specific embeddings and classifier predictions to leverage discriminative information. Besides the usual distribution alignment, further align the input space with a generative model that maps the target input distribution to the source distribution. These previous works can be interpreted as adding additional regularization via auxiliary objectives, and thereby potentially reducing the complexity penalty. Some previous works also optimize the domain-invariant loss in multiple layers. fuse the representations from a bottleneck layer and a classifier layer by a tensor product and minimize the domain divergence based on the aggregated representations. Joint adaptation networks (JADs) minimize the MMD in the last few layers to make the embeddings more transferable. MDM can be seen as a generalization of JADs that minimizes domain divergence in nearly every layer, driven by a strong theoretical motivation. Importantly, minimizing the divergence only in the last few layers could still be suboptimal, since the embeddings may not be sufficiently regularized. We test our theory and algorithm on several standard benchmarks: sentiment analysis (Amazon reviews dataset), digit classification (MNIST, MNIST-M, SVHN) and general object classification (Office-31). In all experiments, we train DANN , which measures the latent divergence via a domain discriminator (Jensen Shannon divergence). A validation set from the source domain is used as an early stopping criterion during learning. In all experiments, we use the Adam optimizer and a progressive training strategy for the discriminator . We primarily consider three types of complexity: number of layers, number of hidden neurons, and inductive bias (CNNs). In all experiments, we retrain each model for 5 times and plot the mean and standard deviation of the target error. For evaluating MDM, we consider three weighting schemes: uniform weights (α i = α 0), linearly decreasing (α i = α 0 −c×i), and exponentially decreasing (α i = α 0 exp(−c×i)) where c ≥ 0. The decreasing weights encourage the network to minimize the latent divergence in the first few layers, where the embedding complexity is low. This may also further restrict the deeper embeddings. More experimental details can be found in Appendix C. Sentiment Classification. We first examine complexity tradeoffs on the Amazon reviews data, which has four domains (books (B), DVD disks (D), electronics (E), and kitchen appliances (K)) with binary labels (positive / negative review). Reviews are encoded into 5000 dimensional feature vectors of unigrams and bigrams. The hypothesis class are multi-layer ReLU networks. We show the on B→K, K→B, B→D, and D→B in Figure 3. To probe the effect of embedding complexity by itself, we fix the predictor class to 4 layers and vary the number of layers of the embedding. In agreement with the in Section 3.2, the target error decreases initially, and then increases as more layers are added to the encoder. Next, we probe the tradeoff when the total number of layers is fixed to 8. The bottom row of Figure 3 shows that there exists an optimal setting for all tasks. For MDM, we optimize alignment in all intermediate layers. The suggest that MDM's performance is comparable to the hypothesis with the optimal division, without tuning the division. The three weighting schemes perform similarly, suggesting that MDM is robust to weight selection. Digit Classification. We next verify our findings on standard domain adaptation benchmarks: MNIST→MNIST-M (M→M-M) and SVHN→MNIST (S→M). We use standard CNNs as the hypothesis class; architecture details are in Appendix C. Number of Layers in Encoder To analyze the effect of the embedding complexity, we augment the original two-layer CNN encoders with 1 to 6 additional CNN layers for M→M-M and 1 to 24 for S→M, leaving other settings unchanged. Figure 4(a) shows the . Again, the target error decreases initially and increase as the encoder becomes more complex. Notably, the target error increases by 19.8% in M→M-M and 8.8% in S→M compared to the optimal case, when more layers are added to the encoder. We also consider the width of hidden layers as a complexity measure, while fixing the depth of both encoder and predictor. The are shown in Figure 4 (b). This time, the decrease in target error is not significant compared to increasing encoder depth. This suggests that depth plays a more important role than width in learning domain-invariant representations. Next, we fix the total number of CNN layers of the neural network to 7 and 26 for M→M-M and S→M, respectively, and optimize the domain-invariant loss in different intermediate layers. The in Figure 4 (c) again show a "U-curve", indicating the existence of an optimal division. Even with fixed total size of the network (H), the performance gap between different divisions can still reach 19.5% in M→M-M and 10.4% in S→M. For MDM, L contains all the augmented CNN layers for M→M-M. For S→M, we sub-sample a CNN layer every four layers to form L. We also observe that MDM with all weighting schemes consistently achieves comparable performance with the best division in S→M and even better performance in M→M-M. To investigate the importance of inductive bias in domain-invariant representations, we replace the CNN encoder by an MLP encoder. The for M→M-M are shown in Figure 5. Comparing to CNNs, which encode invariance via pooling and learned filters, MLPs do not have any inductive bias and lead to worse performance. In fact, the target error with MLP-based domain adaptation is higher than merely training on the source: without an appropriate inductive bias, learning domain invariant representations can even worsen the performance. Object Classification. Office-31 , one of the most widely used benchmarks in domain adaptation, contains three domains: Amazon (A), Webcam (W), and DSLR (D) with 4,652 images and 31 categories. We show for A→W, A→D, W→A, and D→A in Figure 6. To overcome the lack of training data, similar to , we use ResNet-50 pretrained on ImageNet for feature extraction. With the extracted features, we adopt multi-layer ReLU networks as hypothesis class. Again, we increase the depth of the encoder while fixing the depth of the predictor to 2 and show the Figure 6. Even with a powerful feature extractor, the embedding complexity tradeoff still exists. Second, we fix the total network depth to 14 and optimize MDM, with L containing all even layers in the network. MDM achieves comparable performance to the best division for most of the tasks, albeit slightly worse performance in D→A. In this paper, we theoretically and empirically analyze the effect of embedding complexity on the target risk in domain-invariant representations. We find a complexity tradeoff that has mostly been overlooked by previous work. In fact, without carefully selecting and restricting the encoder class, learning domain invariant representations might even harm the performance. We further develop a simple yet effective algorithm to approximately optimize the tradeoff, achieving performance across tasks that matches the best network division, i.e., complexity tradeoff. Interesting future directions of work include other strategies for model selection, and a more refined analysis and exploitation of the effect of inductive bias. A.1 PROOF OF THEOREM 4 Theorem 4. For all f ∈ F and g ∈ G, where λ F G (g) is the best in-class joint risk defined as Proof. We first define the optimal composition hypothesis f * g * with respect to an encoder g to be the hypothesis which minimizes the following error By the triangle inequality for classification error , The second term in the R.H.S of Eq. 13 can be bounded as The third term in the R.H.S of Eq. 13 can be bounded as We investigate the effect of predictor complexity on MNIST→MNIST-M. Follow the procedure in section 6, we augment the original predictor with 1 to 7 additional CNN layers and fix the number of layers in encoder to 4 or vary the hidden width. The are shown in Figure 7. The target error slightly decreases as the number of layers in the predictor increases. Even we augment 7 layers to the predictor, the target error only decrease 0.9% which is nearly ignorable. Therefore, we focus on the embedding complexity in the main paper which is both theoretically and empirically interesting. Target Error Target The learning rate of Adam optimizer is set to 1 × e −3 and the model are trained for 50 epochs. We adopt the original progressive training strategy for discriminator where the weight α for domain-invariant loss in equation is initiated at 0 and is gradually changed to 1 using the following schedule: where p is the training progress linearly changing from 0 to 1. The architecture of the hypothesis and discriminator are as follows: Encoder nn. Linear nn. ReLU nn. Linear nn. ReLU ×n (depends on the number of layers) Predictor nn. Linear nn. ReLU ×n (depends on the number of layers) nn. Linear nn. Softmax Discriminator nn. Linear nn. ReLU nn. Linear nn. ReLU ×5 nn. Linear nn. Softmax The learning rate of Adam optimizer is set to 1 × e −3 and the model are trained for 100 epochs. The weight α for domain-invariant loss in equation is initiated at 0 and is gradually changed to 0.1 using the same schedule in section C.1. The architecture of the hypothesis and discriminator are as follows: Encoder nn. Conv2d(3, 64, kernel size=5) nn. BatchNorm2d nn. MaxPool2d nn. ReLU nn. Conv2d(64, 128, kernel size=5) nn. BatchNorm2d nn. Dropout2d (only added for MNIST→MNIST-M) nn. MaxPool2d nn. ReLU nn. Conv2d(128, 128, kernel size=3, padding=1) nn. BatchNorm2d nn. ReLU ×n (depends on the number of layers) Predictor nn. Conv2d(128, 128, kernel size=3, padding=1) nn. BatchNorm2d nn. ReLU ×n (depends on the number of layers) flatten nn. Linear nn. BatchNorm1d nn. ReLU nn. Linear nn. Softmax Discriminator nn. Conv2d(128, 256, kernel size=3, padding=1) nn. ReLU nn. Conv2d(256, 256, kernel size=3, padding=1) nn. ReLU ×4 Flatten nn. Linear nn. ReLU nn. Linear nn. ReLU nn. Linear nn. Softmax In the hidden width experiments, we treat the architectures above as the pivot and multiply their hidden width with the ratios. We exploit the feature after average pooling layer of the ResNet-50 pretrained on ImageNet for feature extraction. The learning rate of Adam optimizer is set to 3 × e −4 and the model are trained for 100 epochs. The weight α for domain-invariant loss in equation is initiated at 0 and is gradually changed to 1 using the same schedule in section C.1. The architecture of the hypothesis and discriminator are as follows: Encoder nn. Linear nn. ReLU nn. Linear nn. ReLU ×n (depends on the number of layers) Predictor nn. Linear nn. ReLU ×n (depends on the number of layers) nn. Linear nn. Softmax Discriminator nn. Linear nn. ReLU ×6 nn. Linear nn. Softmax In all the experiments, we minimize the divergence in multiple layers by augmenting additional discriminators for each layer-specific representations where the discriminators share the same architecture as the standard setting. For uniform weighting scheme (α i = α 0), α i is set to the normalized same value α in the stand setting. For linear decreasing scheme (α i = α 0 − c × i), α i decreases from α 0 = α to 0 linearly. For exponentially decreasing scheme (α i = α 0 exp(−c × i)), α 0 is set to α and c increases from 0 to 2 linearly.

We study the effect of the embedding complexity in learning domain-invariant representations and develop a strategy that mitigates sensitivity to it.

Represent this text to find a much much longer article about it with say hundreds-thousands of words:

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: We propose a new architecture termed Dual Adversarial Transfer Network (DATNet) for addressing low-resource Named Entity Recognition (NER). Specifically, two variants of DATNet, i.e., DATNet-F and DATNet-P, are proposed to explore effective feature fusion between high and low resource. To address the noisy and imbalanced training data, we propose a novel Generalized Resource-Adversarial Discriminator (GRAD). Additionally, adversarial training is adopted to boost model generalization. We examine the effects of different components in DATNet across domains and languages and show that significant improvement can be obtained especially for low-resource data. Without augmenting any additional hand-crafted features, we achieve new state-of-the-art performances on CoNLL and Twitter NER---88.16% F1 for Spanish, 53.43% F1 for WNUT-2016, and 42.83% F1 for WNUT-2017. Named entity recognition (NER) is an important step in most natural language processing (NLP) applications. It detects not only the type of named entity, but also the entity boundaries, which requires deep understanding of the contextual semantics to disambiguate the different entity types of same tokens. To tackle this challenging problem, most early studies were based on hand-crafted rules, which suffered from limited performance in practice. Current methods are devoted to developing learning based algorithms, especially neural network based methods, and have been advancing the state-of-the-art consecutively BID7 BID23 BID6 BID33. These end-to-end models generalize well on new entities based on features automatically learned from the data. However, when the annotated corpora is small, especially in the low resource scenario BID56, the performance of these methods degrades significantly since the hidden feature representations cannot be learned adequately. Recently, more and more approaches have been proposed to address low-resource NER. Early works BID5 BID24 primarily assumed a large parallel corpus and focused on exploiting them to project information from high-to low-resource. Unfortunately, such a large parallel corpus may not be available for many low-resource languages. More recently, cross-resource word embedding BID9 BID0 BID52 was proposed to bridge the low and high resources and enable knowledge transfer. Although the aforementioned transferbased methods show promising performance in low-resource NER, there are two issues deserved to be further investigated on: 1) Representation Difference -they did not consider the representation difference across resources and enforced the feature representation to be shared across languages/domains; 2) Resource Data Imbalance -the training size of high-resource is usually much larger than that of low-resource. The existing methods neglect such difference in their models, ing in poor generalization. In this work, we present an approach termed Dual Adversarial Transfer Network (DATNet) to address the above issues in a unified framework for low-resource NER. Specifically, to handle the representation difference, we first investigate on two architectures of hidden layers (we use bidirectional long-short term memory (BiLSTM) model as hidden layer) for transfer. The first one is that all the units in hidden layers are common units shared across languages/domains. The second one is composed of both private and common units, where the private part preserves the independent language/domain information. Extensive experiments are conducted to show their advantages over each other in different situations. On top of common units, the adversarial discriminator (AD) loss is introduced to encourage the resource-agnostic representation so that the knowledge from high resource can be more compatible with low resource. To handle the resource data imbalance issue, we further propose a variant of the AD loss, termed Generalized Resource-Adversarial Discriminator (GRAD), to impose the resource weight during training so that low-resource and hard samples can be paid more attention to. In addition, we create adversarial samples to conduct the Adversarial Training (AT), further improving the generalization and alleviating over-fitting problem. We unify two kinds of adversarial learning, i.e., GRAD and AT, into one transfer learning model, termed Dual Adversarial Transfer Network (DATNet), to achieve end-to-end training and obtain the state-of-the-art performance on a series of NER tasks-88.16% F1 for CoNLL-2002 Spanish, 53.43% and 42.83% F1 for WNUT-2016. Different from prior works, we do not use additional hand-crafted features and do not use cross-lingual word embeddings while addressing the cross-language tasks. NER is typically framed as a sequence labeling task which aims at automatic detection of named entities (e.g., person, organization, location and etc.) from free text BID35. The early works applied CRF, SVM, and perception models with handcrafted features BID44 BID42 BID31. With the advent of deep learning, research focus has been shifting towards deep neural networks (DNN), which requires little feature engineering and domain knowledge BID23 BID57. BID7 proposed a feed-forward neural network with a fixed sized window for each word, which failed in considering useful relations between long-distance words. To overcome this limitation, BID6 presented a bidirectional LSTM-CNNs architecture that automatically detects word-and character-level features. BID33 further extended it into bidirectional LSTM-CNNs-CRF architecture, where the CRF module was added to optimize the output label sequence. proposed task-aware neural language model termed LM-LSTM-CRF, where character-aware neural language models were incorporated to extract characterlevel embedding under a multi-task framework. Transfer Learning for NER Transfer learning can be a powerful tool to low resource NER tasks. To bridge high and low resource, transfer learning methods for NER can be divided into two types: the parallel corpora based transfer and the shared representation based transfer. Early works mainly focused on exploiting parallel corpora to project information between the high-and low-resource language BID53 BID5 BID24 BID10. For example, BID5 and BID10 proposed to jointly identify and align bilingual named entities. On the other hand, the shared representation methods do not require the parallel correspondence BID46. For instance, BID9 proposed cross-lingual word embeddings to transfer knowledge across resources. BID52 presented a transfer learning approach based on a deep hierarchical recurrent neural network (RNN), where full/partial hidden features between source and target tasks are shared. BID38 BID39 utilized the Wikipedia entity type mappings to improve low-resource NER. BID2 built massive multilingual annotators with minimal human expertise by using language agnostic techniques. BID36 created a cross-language NER system, which works well for very minimal resources by translate annotated data of high-resource into low-resource. BID8 proposed character-level neural CRFs to jointly train and predict low-and high-resource languages. BID40 proposes a large-scale cross-lingual named entity dataset which contains 282 languages for evaluation. In addition, multi-task learning BID51 BID32 BID45 BID1 BID15 BID28 shows that jointly training on multiple tasks/languages helps improve performance. Different from transfer learning methods, multi-task learning aims at improving the performance of all the resources instead of low resource only. Adversarial Learning Adversarial learning originates from Generative Adversarial Nets (GAN) BID12, which shows impressing in computer vision. Recently, many papers have tried to apply adversarial learning to NLP tasks. BID30 presented an adversarial multi-task learning framework for text classification. BID14 applied the adversarial discriminator to POS tagging for Twitter. BID18 proposed a language discriminator to enable language-adversarial training for cross-language POS tagging. Apart from adversarial discriminator, adversarial training is another concept originally introduced by BID49 BID13 to improve the robustness of image classification model by injecting malicious perturbations into input images. Recently, BID37 proposed a semi-supervised text classification method by applying adversarial training, where for the first time adversarial perturbations were added onto word embeddings. BID54 applied adversarial training to POS tagging. Different from all these adversarial learning methods, our method integrates both the adversarial discriminator and adversarial training in an unified framework to enable end-to-end training. In this section, we introduce DATNet in more details. We first describe a base model for NER, and then discuss two proposed transfer architectures for DATNet. We follow state-of-the-art models for NER task BID23 BID6 BID33, i.e., LSTM-CNNs-CRF based structure, to build the base model. It consists of the following pieces: character-level embedding, word-level embedding, BiLSTM for feature representation, and CRF as the decoder. The character-level embedding takes a sequence of characters in the word as atomic units input to derive the word representation that encodes the morphological information, such as root, prefix, and suffix. These character features are usually encoded by character-level CNN or BiLSTM, then concatenated with word-level embedding to form the final word vectors. On top of them, the network further incorporates the contextual information using BiLSTM to output new feature representations, which is subsequently fed into CRF layer to predict label sequence. Although both of the word-level layer and the character-level layer can be implemented using CNNs or RNNs, we use CNNs for extracting character-level and RNNs for extracting word-level representation. FIG0 shows the the architecture of the base model. Previous works have shown that character features can boost sequence labeling performance by capturing morphological and semantic information BID28. For low-resource dataset to obtain high-quality word features, character features learned from other language/domain may provide crucial information for labeling, especially for rare and out-of-vocabulary words. Character-level encoder usually contains BiLSTM BID23 and CNN BID6 BID33 approaches. In practice, BID47 observed that the difference between the two approaches is statistically insignificant in sequence labeling tasks, but character-level CNN is more efficient and has less parameters. Thus, we use character-level CNN and share character features between high-and low-resource tasks to enhance the representations of low-resource. To learn a better word-level representation, we concatenate character-level features of each word with a latent word embedding as DISPLAYFORM0 ], where the latent word embedding w emb i is initialized with pre-trained embeddings and fixed during training. One unique characteristic of NER is that the historical and future input for a given time step could be useful for label inference. To exploit such a characteristic, we use a bidirectional LSTM architecture BID16 ) to extract contextualized word-level features. In this way, we can gather the information from the past and future for a particular time frame t as follows, DISPLAYFORM1 After the LSTM layer, the representation of a word is obtained by concatenating its left and right context representation as follows, DISPLAYFORM2 To consider the resource representation difference on word-level features, we introduce two kinds of transferable word-level encoder in our model, namely DATNet-Full Share (DATNet-F) and DATNetPart Share (DATNet-P). In DATNet-F, all the BiLSTM units are shared by both resources while word embeddings for different resources are disparate. The illustrative figure is depicted in the FIG0 Different from DATNet-F, the DATNet-P decomposes the BiLSTM units into the shared component and the resource-related one, which is shown in the FIG0. In order to make the feature representation extracted from the source domain more compatible with those from the target domain, we encourage the outputs of the shared BiLSTM part to be resourceagnostic by constructing a resource-adversarial discriminator, which is inspired by the LanguageAdversarial Discriminator proposed by BID18. Unfortunately, previous works did not consider the imbalance of training size for two resources. Specifically, the target domain consists of very limited labeled training data, e.g., 10 sentences. In contrast, labeled training data in the source domain are much richer, e.g., 10k sentences. If such imbalance was not considered during training, the stochastic gradient descent (SGD) optimization would make the model more biased to high resource BID27. To address this imbalance problem, we impose a weight α on two resources to balance their influences. However, in the experiment we also observe that the easily classified samples from high resource comprise the majority of the loss and dominate the gradient. To overcome this issue, we further propose Generalized Resource-Adversarial Discriminator (GRAD) to enable adaptive weights for each sample (note that the sample here means each sentence of resource), which focuses the model training on hard samples. To compute the loss of GRAD, the output sequence of the shared BiLSTM is firstly encoded into a single vector via a self-attention module BID3, and then projected into a scalar r via a linear transformation. The loss function of the resource classifier is formulated as: DISPLAYFORM0 where I i∈D S, I i∈D T are the identity functions to denote whether a sentence is from high resource (source) and low resource (target), respectively; α is a weighting factor to balance the loss contribution from high and low resource; the parameter (1 − r i) γ (or r γ i) controls the loss contribution from individual samples by measuring the discrepancy between prediction and true label (easy samples have smaller contribution); and γ scales the contrast of loss contribution from hard and easy samples. In practice, the value of γ does not need to be tuned much and usually set as 2 in our experiment. Intuitively, the weighting factors α and (1 − r i) γ reduce the loss contribution from high resource and easy samples, respectively. Note that though the resource classifier is optimized to minimize the resource classification error, when the gradients originated from the resource classification loss are back-propagated to the other model parts than the resource classifier, they are negated for parameter updates so that these bottom layers are trained to be resource-agnostic. The label decoder induces a probability distribution over sequences of labels, conditioned on the word-level encoder features. In this paper, we use a linear chain model based on the first-order Markov chain structure, termed the chain conditional random field (CRF) BID22, as the decoder. In this decoder, there are two kinds of cliques: local cliques and transition cliques. Specifically, local cliques correspond to the individual elements in the sequence. And transition cliques, on the other hand, reflect the evolution of states between two neighboring elements at time t − 1 and t and we define the transition distribution as θ. Formally, a linear-chain CRF can be written as p(y|h 1: DISPLAYFORM0 W yt h t, where Z(h 1:T) is a normalization term and y is the sequence of predicted labels as follows: y = y 1:T. Model parameters are optimized to maximize this conditional log likelihood, which acts as the objective function of the model. We define the loss function for source and target resources as follows, S = − i log p(y|h 1:T), T = − i log p(y|h 1:T). So far our model can be trained end-to-end with standard back-propagation by minimizing the following loss: DISPLAYFORM0 Recent works have demonstrated that deep learning models are fragile to adversarial examples BID13. In computer vision, those adversarial examples can be constructed by changing a very small number of pixels, which are virtually indistinguishable to human perception BID43. Recently, adversarial samples are widely incorporated into training to improve the generalization and robustness of the model, which is so-called adversarial training (AT) BID37. It emerges as a powerful regularization tool to stabilize training and prevent the model from being stuck in local minimum. In this paper, we explore AT in context of NER. To be specific, we prepare an adversarial sample by adding the original sample with a perturbation bounded by a small norm to maximize the loss function as follows: DISPLAYFORM1 where Θ is the current model parameters set. However, we cannot calculate the value of η exactly in general, because the exact optimization with respect to η is intractable in neural networks. Following the strategy in BID13, this value can be approximated by linearizing it as follows, DISPLAYFORM2 where can be determined on the validation set. In this way, adversarial examples are generated by adding small perturbations to the inputs in the direction that most significantly increases the loss function of the model. We find such η against the current model parameterized by Θ, at each training step, and construct an adversarial example by x adv = x + η x. Noted that we generate this adversarial example on the word and character embedding layer, respectively, as shown in the FIG0 (b) and 1(c). Then, the classifier is trained on the mixture of original and adversarial examples to improve the generalization. To this end, we augment the loss in Eqn. 2 and define the loss function for adversarial training as: DISPLAYFORM3 where (Θ; x), (Θ; x adv) represents the loss from an original example and its adversarial counterpart, respectively. Note that we present the AT in a general form for the convenience of presentation. For different samples, the loss and parameters should correspond to their counterparts. For example, for the source data with word embedding w S, the loss for AT can be defined as follows, AT = (Θ; w S) + (Θ; w S,adv) with w S,adv = w S + η w S and = GRAD + S. Similarly, we can compute the perturbations η c for char-embedding and η w T for target word embedding. In order to evaluate the performance of DATNet, we conduct the experiments on following widely used NER datasets: CoNLL-2003 English NER BID20, CoNLL-2002 Spanish & Dutch NER BID19, WNUT-2016 English Twitter NER (. The statistics of these datasets are described in TAB1 . We use the official split of training/validation/test sets. Since our goal is to study the effects of transferring knowledge from high-resource dataset to low-resource dataset, unlike previous works BID7 BID6 BID52 to append one-hot gazetteer features to the input of the CRF layer, and the works BID41 BID25 BID1 to introduce orthographic feature as additional input for learning social media NER in tweets, we do not experiment with hand-crafted features and only consider words and characters embeddings as the inputs of our model. To be noted, we used only train set for model training for all datasets except the WNUT-2016 NER dataset. Since in this dataset, all the previous studies merged the training and validation sets together for training, we followed the same way for fair comparison. In addition to the CoNLL and WNUT datasets, we also experiment on the cross-language named entity dataset described in BID40, which contains datasets for 282 languages, to evaluate our methods and investigate the transferability of different linguistic families and branches in both low-and high-resource scenarios. We choose 9 languages in our experiment, where Galician (gl), West Frisian (fy), Ukrainian (uk) and Marathi (mr) are target languages, the corresponding source languages are Spanish (es), Dutch (nl), Russian (ru) and Hindi (hi), and Arabic (ar) is also a source language, which is from different linguistic family. Following the setting in BID8, we also simulate the low-and high-resource scenarios by creating 100 and 10,000 sentences split for training target language datasets, respectively. Then we create 1,000 sentences split for validation and test, respectively. For source languages, we create 10,000 sentence split for training only. For high-resource scenario, we only conduct experiments on Galician (gl-high) and Ukrainian (uk-high).The list of selected datasets are described in TAB2. BID28. For the named entity datasets selected from BID40, we use 300-dimensional pre-trained word embeddings trained by fastText package 3 on Wikipedia BID4, and the 30-dimensional randomly initialized character embeddings are used for all the datasets. We set the filter number as 20 for char-level CNN and the dimension of hidden states of the word-level LSTM as 200 for both base model and DATNet-F. For DATNet-P, we set 100 for source, share, and target LSTMs dimension, respectively. Parameters optimization is performed by Adam optimizer BID21 with gradient clipping of 5.0 and learning rate decay strategy. We set the initial learning rate of β 0 = 0.001 for all experiments. At each epoch t, learning rate β t is updated using β t = β 0 /(1 + ρ × t), where ρ is decay rate with 0.05. To reduce over-fitting, we also apply Dropout BID48 to the embedding layer and the output of the LSTM layer, respectively. In this section, we compare our approach with state-of-the-art (SOTA) methods on CoNLL and WNUT benchmark datasets. In the experiment, we exploit all the source data (i.e., CoNLL-2003 English NER) and target data to improve performance of target tasks. The averaged with standard deviation over 10 repetitive runs are summarized in TAB3, and we also report the best on each task for fair comparison with other SOTA methods. From , we observe that incorporating the additional resource is helpful to improve performance. TAB4 summarizes the of our methods under different cross-language transfer settings as well as the comparison with BID8. In this experiment, we study the transferability between languages not only from same linguistic family and branch, but also from different linguistic families or branches. According to the , DATNets outperform the transfer method of BID8 for both low-and high-resource scenarios within the same linguistic family and branch (i.e., in-family in-branch) transfer case. We also observe that: 1) For the low-resource scenario, transfer learning is significantly helpful for improving the performance of target datasets within both same and different linguistic family or branch (i.e., in/cross-family in/cross-branch) transfer cases, while the improvements are more prominent under the in-family in-branch case. 2) For the high-resource scenario, say, when the target language data is sufficient, the improvements of transfer learning are not very distinct compared with that for low-resource scenario under in-family in-branch case. We also find that there is no effect by transferring knowledge from Arabic to Galician and Ukrainian. We suspect that it is caused by the great linguistic differences between source and target languages, since, for example, Arabic and Galician are from totally different linguistic families. In this section, we investigate on improvements with transfer learning under multiple low-resource settings with partial target data. To simulate a low-resource setting, we randomly select subsets of target data with varying data ratio at 0. 05, 0.1, 0.2, 0.4, 0.6, and 1.0. For example, 20, 748 training tokens are sampled from the training set under a data ratio of r = 0.1 for the dataset CoNLL-2002 Spanish NER (Cf. TAB1). The for cross-language and cross-domain transfer are shown in FIG2 (a) and 2(b), respectively, where we compare the with each part of DATNet under various data ratios. From those figures, we have the following observations: 1) both adversarial training and adversarial discriminator in DATNet consistently contribute to the performance improvement; 2) the transfer learning component in the DATNet consistently improve over the base model and the improvement margin is more substantial when the target data ratio is lower. For example, when the data ratio is 0.05, DATNet-P model outperforms the base model by more than 4% absolutely in F1-score on Spanish NER and DATNet-F model improves around 13% absolutely in F1-score compared to base model on WNUT-2016 NER.In the second experiment, we further investigate DATNet on the extremely low resource cases, e.g., the number of training target sentences is 10, 50, 100, 200, 500 and 1,000. The setting is quite challenging and fewer previous works have studied before. The are summarized in TAB5. We have two interesting observations 5: 1) DATNet-F outperforms DATNet-P on cross-language transfer when the target resource is extremely low, however, this situation is reversed when the target dataset size is large enough (here for this specific dataset, the threshold is 100 sentences); 2) DATNet-F is always superior to DATNet-P on cross-domain transfer. For the first observation, it is because DATNet-F with more shared hidden units is more efficient to transfer knowledge than DATNet-P when data size is extremely small. For the second observation, because cross-domain transfer are in the same language, more knowledge is common between the source and target domains, requiring more shared hidden features to carry with these knowledge compared to cross-language transfer. Therefore, for cross-language transfer with an extremely low resource and cross-domain transfer, we suggest using DATNet-F model to achieve better performance. As for cross-language transfer with relatively more training data, DATNet-P model is preferred. In the proposed DATNet, both GRAD and AT play important roles in low resource NER. In this experiment, we further investigate how GRAD and AT help transfer knowledge across language/domain. In the first experiment 6, we used t-SNE BID34 to visualize the feature distribution of BiLSTM outputs without AD, with normal AD (GRAD without considering data imbalance), and with the proposed GRAD in FIG3. From this figure, we can see that the GRAD in DATNet makes the distribution of extracted features from the source and target datasets 5 For other tasks/languages we have the similar observation, we only report CoNLL-2002 Spanish and WNUT-2016 Twitter due to the page limit. 6 We used data ratio ρ = 0.5 for training model and randomly selected 10k testing data for visualization. much more similar by considering the data imbalance, which indicates that the outputs of BiLSTM are resource-invariant. To better understand the working mechanism, TAB6 further reports the quantitative performance comparison between models with different components. We observe that GRAD shows the stable superiority over the normal AD regardless of other components. There are no always winner between DATNet-P and DATNet-F on different settings. DATNet-P architecture is more suitable to cross-language transfer while DATNet-F is more suitable to cross-domain transfer. From the previous , we know that AT helps enhance the overall performance by adding perturbations to inputs with the limit of = 5, i.e., η 2 ≤ 5. In this experiment, we further investigate how target perturbation w T with fixed source perturbation w S = 5 in AT affects knowledge transfer and the on Spanish NER are summarized in TAB7. The generally indicate that less training data require a larger to prevent over-fitting, which further validates the necessity of AT in the case of low resource data. Finally, we analyze the discriminator weight α in GRAD and are summarized in TAB8. From the , it is interesting to find that α is directly proportional to the data ratio ρ, basically, which means that more target training data requires larger α (i.e., smaller 1 − α to reduce training emphasis on the target domain) to achieve better performance. In this paper we develop a transfer learning model DATNet for low-resource NER, which aims at addressing two problems remained in existing work, namely representation difference and resource data imbalance. We introduce two variants of DATNet, DATNet-F and DATNet-P, which can be chosen for use according to the cross-language/domain user case and the target dataset size. To improve model generalization, we propose dual adversarial learning strategies, i.e., AT and GRAD. Extensive experiments show the superiority of DATNet over existing models and it achieves new state-of-the-art performance on CoNLL NER and WNUT NER benchmark datasets.

We propose a new  architecture termed Dual Adversarial Transfer Network (DATNet) for addressing low-resource Named Entity Recognition (NER) and achieve new state-of-the-art performances on CoNLL and Twitter NER.

Represent the short summary for finding the corresponding scientific article (long)

We provide insightful understanding of sequence-labeling NER and propose to use two types of cross structures, both of which bring theoretical and empirical improvements.