Question:
represent the scientific article to find a short tldr

For example, 'The field of deep learning has been craving for an optimization method that shows outstanding property for both optimization and generalization. We propose a method for mathematical optimization based on flows along geodesics, that is, the shortest paths between two points, with respect to the Riemannian metric induced by a non-linear function. In our method, the flows refer to Exponentially Decaying Flows (EDF), as they can be designed to converge on the local solutions exponentially. In this paper, we conduct experiments to show its high performance on optimization benchmarks (i.e., convergence properties), as well as its potential for producing good machine learning benchmarks (i.e., generalization properties). Due to recent progress in the field of machine learning, it becomes more and more important to develop and sophisticate methods of solving hard optimization problems. At the same time, in this field, such methods are additionally required to elicit decent generalization performance from statistical models. An efficient method of mathematical optimization, however, does not always produce sufficient generalization properties, since these are involved with two distinct mathematical problems; The former is to find one of the solutions which minimize a given (possibly non-convex) objective function, and the latter is to adjust parameters so that a statistical estimator achieves its best . To address such a hard issue, we introduce a new mathematical perspective on optimization, and develop a method for machine learning based on this perspective. We then empirically show its rapid convergence rate and high compatibility with deep learning techniques, as well as good statistical properties. In this field, many optimization methods have been proposed and modified so that they fit specific problems or models. One of the current standard methods is the gradient descent method. The method tends to converge slowly in general optimization problems. However, with various specific techniques, such as mini-batch training and batch normalization BID9 ), it has been found to be efficient for state-of-the-art purposes in the field of deep learning. Another class of methods that are now becoming popular and standard is adaptive methods, such as AdaGrad BID6 ) and Adam . Compared to the gradient descent method, these methods have been shown to improve convergence rates with almost the same computational cost as the gradient descent method, but are reported to in poor statistical outcomes in some cases of machine learning BID16 ).Other class of methods that have been thoroughly studied in the theory of mathematical optimization is second-order methods, such as the Newton method and the Gauss-Newton method. These methods possess great convergence properties, and in particular, have a potential to overcome plateau's problems BID5 ). Furthermore, when it comes to applications in stochastic settings, the method based on the Gauss-Newton Matrix (or Fisher information Matrix) is shown to asymptotically attain the best statistical , which is called Fisher efficiency (see BID0). Despite these attractive characteristics, the methods have not yet been spotlighted in the field of machine learning due to several severe drawbacks; They suffer from high computational cost in general and their useful properties are no longer guaranteed in practical settings (see Section 12 in BID12). One of the continuously developing second-order methods in this field, K-FAC BID1, BID7 ), successfully produced high convergence rate empirically with relatively low computational cost. However, it still requires much effort to become compatible with some deep learning techniques. In addition, it is unclear whether the method has advantages in generalization performance. In our approach, by introducing a Riemannian metric induced by non-linear functions, we constitute dynamical systems which describe motions along the shortest route from arbitrary initial points to the zeros of non-linear functions on the corresponding Riemannian manifold, that is, geodesic with respect to the Riemannian metric. One of the remarkable characteristics of our approach is that it enables us to flexibly design flows of such dynamical systems to control convergence rates. The for the flows are then applicable to mathematical optimization problems, in particular, with deep neural network (DNN) models. In this paper, after providing mathematical ground of our methods, we experimentally demonstrate their performance in various aspects, from convergence rates to statistical properties. We start by establishing some essential properties of dynamics which are effective for the analysis of non-linear equations. Let F: R N → R N be a smooth function and J be the Jacobian with variable w ∈ R N, that is, J = ∂F/∂w. In this section, we deal with the well-posed case that there exists a connected closed subset Ω ⊂ R N, where J is regular and the equation has a unique solution ξ. Therefore, the positive matrix G = J T J induces a Riemannian metric g on Ω and (Ω, g) becomes a Riemannian manifold under some appropriate conditions. Let us then consider time evolution of variable w on this manifold. Our main purpose in this section is to study the characteristics of dynamical systems in which w(t) moves on geodesics between any point in Ω and ξ with respect to the metric g. Let L be a Lagrangian given by DISPLAYFORM0 with v = dw/dt (also written asẇ). The Euler-Lagrange equation for L is then expressed as dp dt DISPLAYFORM1 with momentum vector p = Gv. If the boundary condition at two points in Ω, w(t 0) = w 0, w(t 1) = w 1, is imposed on, a geodesic between w 0 and w 1 is obtained as the solution. In contrast, if we give an appropriate initial condition, w describes the motion along the geodesic from w 0 to w 1. In fact, the following statement holds; Theorem 2.1. Let w 0 be an arbitrary point in Ω and (w(t), p(t)) be a solution of equation FORMULA1 with the following initial condition; DISPLAYFORM2 Then w satisfies F w(t) = (1 − t)F w 0, for t ∈. In particular, w(t) passes through the point which is a solution of non-linear equation DISPLAYFORM3 We briefly describe the outline of the proof for the statement above. Throughout this paper, we regard functions of w as those of t in the trivial way; for instance, F (t) = F (w(t)). Note that p can be expressed as p = J T dF/dt. Then using the Beltrami identity for with the initial condition above leads to the equation DISPLAYFORM4 where F 0 = F. Thus, a closed form expression is obtained as DISPLAYFORM5 which gives F = 0 as asserted in Theorem 2.1. Now we take a different expression that the coefficient (1 − t) in is replaced by a different monotonically decreasing function, that is, DISPLAYFORM6 where ρ denotes a monotonically decreasing smooth function from (t 0, t 1) onto. Then, we give the following differential equation whose solution is of the closed form; DISPLAYFORM7 A motion described by this differential equation differs from the one that is described by, but these two motions are along the same geodesic. Theorem 2.2. Let w 0 ∈ Ω be an arbitrary point. The differential equation DISPLAYFORM8 with an initial condition w 0 = w(t 0) has a unique solution that satisfies F (w(t 1)) = 0. Furthermore, the orbit under flow f defined by f (w 0, t) = w(t) coincides with that of the geodesic equation.Note that since equation FORMULA8 is equivalent to equation FORMULA7, the orbit is invariant under coordinate transformations. With respect to the choice of ρ, the end point t 1 can be set as ∞ under some appropriate conditions and Theorem 2.2 still holds in the sense that DISPLAYFORM9 In particular, if we set ρ(t) = e −t, then χ(t) = −1 and F can be represented as F (t) = e −t F 0, so that the convergence rate of F is exponential. Definition 2.3. Let w be a solution of the differential equation DISPLAYFORM10 with an initial condition w 0 = w(t 0). The flow f (w 0, t) = w(t) is called an exponentially decaying flow (EDF) of non-linear equation F (w) = 0.For the end of this section, we present a connection between EDF and the classical Newton method. If we apply the Euler method to the differential equation with step size t, the corresponding iteration step can be written as DISPLAYFORM11 which recovers the Newton method with step size η i = t · χ(τ i). Consider smooth functions ϕ: DISPLAYFORM0 In this section, we develop a method based on EDF for a mathematical optimization problem of the form DISPLAYFORM1 In this section, the area Ω ⊂ R N is supposed to be a compact subset where there exists no stationary point except for a minima. DISPLAYFORM2 An example of such problems is least squares one in which L is given by DISPLAYFORM3 In this case, F = ϕ. In particular, if M = N and a minimal value of loss function L = L • ϕ is zero, the optimization problem is equivalent to solving non-linear equation F (w) = 0.For optimization problems, we set up a standard equation that the gradient of loss function is zero, that is, ∇ L(w) = 0. Note that the gradient can be expressed as ∇ L(w) = J T ϕ F (w) with Jacobian J ϕ = ∂ϕ/∂w. Applying Theorem 2.2, we obtain the differential equation DISPLAYFORM4 where H is the Hessian of loss function L with respect to w. Since second order differentiation is typically computationally heavy, we seek other equations which possess almost the same property as especially in terms of asymptotic behavior. Let us decompose momentum Hẇ as DISPLAYFORM5 where J F denotes the Jacobian of F, and G is a symmetric matrix defined by DISPLAYFORM6 with Hessian matrix H L of L with respect to ϕ. We then consider the following equation instead of FORMULA0; DISPLAYFORM7 This equation no longer describes the motion along the geodesic related to in general. However, if M = N and J ϕ is invertible, then w moves on another geodesic with respect to a different metric DISPLAYFORM8 In addition, if ρ(t) = e −t, F converges exponentially, which implies that ∇ L = J ϕ F also converges exponentially in as well as in FORMULA0. In general cases, if a condition that DISPLAYFORM9 is satisfied, then F converges to 0. This shows that in the neighborhood of solution ξ of equation ∇ L = 0, the momentum Gẇ sufficiently approximates Hẇ by. Definition 3.1. The flow given by DISPLAYFORM10 is referred to EDF of type H and written as EDF-H. Similarly, the flow given by DISPLAYFORM11 is referred to EDF of type G and written as EDF-G. Like second order methods, in EDF-based methods, matrix inversion has to be carried out, which requires expensive computational cost, particularly in large systems. Moreover, we often encounter rank deficient matrices in optimization problems. To deal with rank deficiency, in general, we need pseudo-inverse matrices which are more computationally heavy. Therefore, instead of the inverse of matrix A = G, H, for fixed v ∈ R M, we consider a projection which maps r = arg min DISPLAYFORM0 One of the basic methods to construct such a projection for indefinite symmetric systems is the minimum residual method BID13 ), which requires only the matrix multiplication. Therefore, for numerical computations, we use the following differential equation approximated by the k-th order Krylov subspace; DISPLAYFORM1 k is a hyperparameter that interpolates between the gradient method and the method based on or. In fact, in the case that k = 1, equation FORMULA1 has the form DISPLAYFORM2 which reduces to a kind of gradient methods, but the coefficient c conveys information about G or H unlike the standard gradient methods. Next, similar to the Levenberg-Marquardt algorithm, we modify equation FORMULA0 by adding a damping factor to G in order that the method becomes more stable. So, we set DISPLAYFORM3 where λ is a continuous positive function of t. Then, we take the same approximation as FORMULA1 with A = G+λI. The damping factor λ plays several important roles in solving practical problems. First, it has solutions well-behaved near singularities. Even in the case that G is not invertible, equation FORMULA1 can be defined and solved unlike. If we choose λ such that it approaches to 0 as rapidly as the gradient in, the asymptotic behavior of FORMULA1 is almost the same as that of FORMULA0. Particularly, in the case that χ = −1, we set λ = a J T ϕ F b with a, b > 0, so that the convergence rate of stays exponential. Second, the damping factor makes the method compatible with stochastic approaches such as mini-batch training, in deep learning models. Since the orbit of FORMULA1 is affected by the gradient due to the damping factor λ, the method based on this equation could take advantage of stochastic approaches as other gradient methods do. (For implementation of the algorithm, see Appendix A.) Finally, to accelerate the EDF-based methods, it is sometimes effective to change equation FORMULA0 into a second-order differential equation, particularly in the case in which the approximation method with k is applied. Specifically, we take the equation DISPLAYFORM4 where κ is a real-valued function of t. The convergence properties of the methods are controlled by the following differential equation; DISPLAYFORM5 There are two ways of determining κ. The first one is to set κ = α with constant α (W1), which leads to a similar scheme to the momentum gradient decent method. The other one is to set κ(t) = αt −1 (W2). In this setting, the equation can be discretized in a similar way as described in BID15, which is analogous to the Nesterov's acceleration scheme. In this section, we present how optimization problems are set up in the field of deep learning. Let x = {x j} n−1 j=0 and y = {y j} n−1 j=0 denote training datasets of input and output, respectively, where n is the size of dataset. Let d x and d y be dimensions of input data and output data, respectively. We write ϕ nn for neural networks with parameters w ∈ R N, and define ϕ by the direct sum of vectors {ϕ j} given by ϕ j (w) = ϕ nn (x j, w), that is, ϕ = ⊕ϕ j. Note that M = n × d y in this case. Then finding a minima of a given loss function is proposed as a standard optimization problem to train networks. For the image classification tasks, there are two typical cases of setting loss functions. In the first case, the loss is set as. As already mentioned, in this case, F = ϕ and H L = I. In the second case, the loss is given by cross entropy with softmax function, that is, DISPLAYFORM0 where θ denotes the softmax function and θ j = θ(ϕ j). In this case, F is expressed by the direct sum such that F = ⊕F j with DISPLAYFORM1 where s j denotes a sum of all elements in vector y j for each j. Note that if each y j is given as a probability vector, then s j = 1. Moreover, H L is expressed as H L = ⊕H j with DISPLAYFORM2 where ⊗ denotes the outer product. In both cases, the loss functions take the minimum value 0 if and only if F = 0. In this study, we conducted following three groups of experiments; First, we examined optimization performance of the EDF-based methods on a data-fitting problem with a simple convolutional neural network model. Second, we tested both optimization and generalization performance in standard settings of classification tasks in which we employed residual network models and CIFAR-10/100 datasets. Third, we incorporated some techniques of data augmentation and regularization in the training into our experiment as we would like to measure effectiveness of our methods in more practical settings. The primary purpose of the experiments in this paper is not to pursue the state-of-the-art performance, but to explore how the statistical pertain to those of optimization when the loss functions are minimized. Therefore, we tuned hyperparameters of each method in accordance with the optimization benchmark. It should be noted that the conjecture concerning non-convex optimization problems in the field of deep learning BID2, BID4 ) is still an open problem (studied for linear cases in BID3, BID10). Hence, for experiments in this paper, we do not discuss whether each optimization method actually reaches to a global solution. We evaluated convergence performance of EDF-based methods (type G and type H) on a data-fitting problem of CIFAR-10, that is, full-batch training in the context of deep learning. The model we employed in these experiments was the convolutional neural network that consisted of two convolution filters with rectified linear units and max pooling layers, followed by two fully connected layers. For EDF-based methods, the step size was fixed to 1.0, and no damping factors were used. In addition, the acceleration technique derived from W2 of was adapted, since W2 achieved better performance than W1. A similar experiment with a different type of second-order methods was conducted in BID14.First, we examined change in convergence performance of EDF-G depending on hyperparameter k, the order of Krylov subspace. The are illustrated in the left-hand side of FIG0. The presented trend is consistent with the theoretical fact that k interpolates between the gradient method (for small k) and the method based on dynamics (for large k). In other words, when k is small, the method is similar to a gradient method, which converges slow, but as k becomes larger, the method leads to better approximation of the inverse matrix, which gives a rapidly decaying flow. Next, we compared EDF-G (k = 1, 30) with EDF-H and other standard optimizers in deep learning: gradient descent methods with Polyaks momentum scheme (Momentum) and Nesterov's acceleration scheme (NAG), and Adam. The step sizes for Momentum, NAG, and Adam were fixed to 0.01, 0.001, and 0.001, respectively. The right-hand side of FIG0 shows that both EDF-based methods made the loss decrease more rapidly than other standard optimizers. Even when EDF-based methods were reduced to the gradient methods at k = 1, EDF-G outperformed standard optimizers. The figure also presents the difference between EDF-G and EDF-H. While their convergence rates around extremes were almost the same, the overall convergence rates of EDF-G were better than that of EDF-H.As has been found, second-order-methods on full-batch training converge to the solution within a few iterations. However, with such a training, the quality of generalization becomes worse compared to the time when stochastic approach with a mini-batch of small size is taken. Like other secondorder-methods, EDF also suffers from a negative impact of full-batch training on generalization performance, though setting the hyperparameter k small makes EDF be compatible with stochastic approaches and take their advantages (see Appendix B). In the following experiments, we compared both optimization and generalization performance of EDF-G with other methods, momentum stochastic gradient decent method (MSGD) and Adam on classification tasks for CIFAR-10/100. The experiments were conducted, employing residual network models with batch normalization (Resnet-56 for CIFAR-10 and Resnet-110 for CIFAR-100), and working with a mini-batch of size 250.For CIFAR-100, during pre-investigation of the dataset, we found that 9 pairs of data, each of which has the same image but different labels, contaminated the neural network and might have had a non-negligible influence on both optimization and generalization performance (See Appendix C). Therefore, in our experiments on CIFAR-100, we excluded them from the training dataset and used the rest of the data (size = 49982).At the end of each epoch, we computed training loss in the following manner. First, the loss was defined as FORMULA1 where n is the total number of data. Second, to calculate the statistics for batch normalization in the loss, as described in BID9, we adopted so-called "inference mode" in which moving averages of mean and variance over mini-batches are used for batch-normalization. For EDF, we tested the case k = 1 and k = 2, with the damping factor set as a = b = 1. The momentum coefficient was fixed to α = 0.9 for the W1 acceleration (see 25).For each of the tasks, we ran tests on 10 different learning rates; for EDF between 0.1 and 2.0, for Momentum SGD between 0.1 and 10.0, for Adam between 0.0001 and 0.1. We then chose the one with the best convergence performance for each optimizer. For EDF with W2, to obtain stability in convergence, the learning rate was set to 0.2 times that of the initial values at the end of the 20-th epoch. Such a change in learning rate in the middle of optimization did not bring advantages to either optimization or generalization performances for other methods including EDF with W1 (see Appendix D).The are presented in Figures 2 and 3. As shown in the figures, with respect to the optimization performance, EDF reached an optimal solution with smaller error at higher convergence rate than Adam and Momentum SGD, even when k = 1, 2. Moreover, EDF overall attained better generalization performance than other methods. For classification tasks, generalization performance often improves by adopting several techniques, such as data augmentation and regularization. In this group of experiments, employing the data augmentation based on random horizontal flip and shift and the L 2 regularization, we conducted comparisons between EDF and other methods that are similar to those in the previous sections. When adopting these techniques, rate decay scheme has as large impact on optimization performance as learning rate itself. Because effectiveness of each rate decay scheme much depends on optimization methods, it is almost impossible to find the best scheme that is applicable to all the methods. For this reason, for MSGD and Adam, as initial learning rates, we chose one of the standard rates for MSGD and Adam, 0.1 and 0.001, respectively, and at the ends of the 100-th, 140-th, 180-th epochs, reset the rates to 0.1 times the rates at the moment. For EDF, we ran tests on two different rates 0.5 and 0.75, the optimal rates found in the experiments of Section 6.2, and reset to 0.2 times those of the initial values, only at the end of the 100-th epoch, in order to demonstrate performance of the EDF more clearly. Among the obtained with EDF, we chose the ones with the best optimization performance. The of the comparison is presented in Figures 4 and 5. As can be seen, we found a condition in which EDF achieved better performance than other methods on optimization while achieving sufficient levels of generalization. Obtaining good statistical from limited available data is a critical goal in machine learning. To reach this goal, while developing an effective model is an essential approach, eliciting the best performance from the fixed model through optimization is important as well. In our study, to examine the performance of our optimization methods, Exponentially Decaying Flows (EDF) based methods, we explored their generalization properties pertaining to of optimization. Our experiments showed that EDF-based methods are more likely to achieve optimal solutions which generalize the test data well than other standard optimizers are. Therefore, EDF-based methods are considered to be optimization methods that have a high potential in their application to various tasks, and thus, are worthwhile to be sophisticated through future studies. In terms of computation of the EDF-based methods with GPU, the Jacobian-vector-product can be carried out at almost the same cost as the gradient of loss function. In fact, multiplying a vector by Jacobian and its transposed matrix (written as R-op and L-op, respectively) are implemented in combination with gradients of scholar functions. For the psuedo-code for update scheme of the EDF-G with L/R-op, refer to Algorithm 1, and Algorithm 2 in particular case that k = 1.Algorithm 1: Update scheme for EDF-G with non-preconditioned MINRES Input: FIG3 shows the of experiments using EDF for simple examples that compare a full-batch training with stochastic trainings. In this example, the convolutional network similar to that used in Section 6.1 was employed on MNIST. The curve labeled as "EDF F" depicts the of Full-batch training per step, and those labeled as "EDF S" illustrate the of stochastic trainings per epoch with a mini-batch of size 500. DISPLAYFORM0 Let us divide the dataset {x, y} = {x j, y j} n−1 j=0 into distinct subsets of size p such that DISPLAYFORM0 Then, gradients and Jacobian-vector-multiplications are calculated as DISPLAYFORM1 where ϕ (i) and F (i) are the subcomponents of ϕ and F, respectively, corresponding to the decomposition above. Thus, for a fixed k, if we set p as the same size as a mini-batch, then the computational cost per step of a full-batch training becomes almost the same as that per epoch of a mini-batch training. The dataset of CIFAR-100 includes 9 pairs of irregular data, each of which has the same image but different labels. These data are enumerated in Figure 7. For instance, the 8393-rd and the 36874-th images are the same, but they have different labels, "girl (class 35)" and "baby (class 2)." Such pairs contaminate the training process. In fact, in our experiment, when the network model was optimized with the full dataset, one of the images in each 9 pairs above could not be classified correctly, which ed in stagnated training accuracy at 99.982 %. Moreover, generalization also deteriorated when irregular data were contained. For the details of these , see Figure 8. D PERFORMANCES WITH THE RATE DECAY SCHEME Figure 9 and FIG0 present the optimization and generalization performance of each optimizer when the rate decay scheme was adopted to the experiment with the same setting as in Figure 2 and Figure 3. The rate decay scheme was that the learning rate was set to 0.2 times that of the initial values at the end of the 20-th epoch.' should have a representation like 'Introduction of a new optimization method and its application to deep learning.' but very far from 'We prove that for a large class of functions f there exists an interval certified robust network approximating f up to arbitrary precision.'.

Question: Domain adaptation tackles the problem of transferring knowledge from a label-rich source domain to an unlabeled or label-scarce target domain. Recently domain-adversarial training (DAT) has shown promising capacity to learn a domain-invariant feature space by reversing the gradient propagation of a domain classifier. However, DAT is still vulnerable in several aspects including training instability due to the overwhelming discriminative ability of the domain classifier in adversarial training, restrictive feature-level alignment, and lack of interpretability or systematic explanation of the learned feature space. In this paper, we propose a novel Max-margin Domain-Adversarial Training (MDAT) by designing an Adversarial Reconstruction Network (ARN). The proposed MDAT stabilizes the gradient reversing in ARN by replacing the domain classifier with a reconstruction network, and in this manner ARN conducts both feature-level and pixel-level domain alignment without involving extra network structures. Furthermore, ARN demonstrates strong robustness to a wide range of hyper-parameters settings, greatly alleviating the task of model selection. Extensive empirical validate that our approach outperforms other state-of-the-art domain alignment methods. Additionally, the reconstructed target samples are visualized to interpret the domain-invariant feature space which conforms with our intuition. Deep neural networks have gained great success on a wide range of tasks such as visual recognition and machine translation . They usually require a large number of labeled data that can be prohibitively expensive to collect, and even with sufficient supervision their performance can still be poor when being generalized to a new environment. The problem of discrepancy between the training and testing data distribution is commonly referred to as domain shift . To alleviate the effect of such shift, domain adaptation sets out to obtain a model trained in a label-rich source domain to generalize well in an unlabeled target domain. Domain adaptation has benefited various applications in many practical scenarios, including but not limited to object detection under challenging conditions , cost-effective learning using only synthetic data to generalize to real-world imagery , etc. Prevailing methods for unsupervised domain adaptation (UDA) are mostly based on domain alignment which aims to learn domain-invariant features by reducing the distribution discrepancy between the source and target domain using some pre-defined metrics such as maximum mean discrepancy . proposed to achieve domain alignment by domainadversarial training (DAT) that reverses the gradients of a domain classifier to maximize domain confusion. Having yielded remarkable performance gain, DAT was employed in many subsequent UDA methods . Even so, there still exist three critical issues of DAT that hinder its performance: as the domain classifier has high-capacity to discriminate two domains, the unbalanced adversarial training cannot continuously provide effective gradients, which is usually overcome by manually adjusting the weights of adversarial training according to specific tasks; DAT-based methods cannot deal with pixel-level domain shift ; the domain-invariant features learned by DAT are only based on intuition but difficult to interpret, which impedes the investigation of the underlying mechanism of adversarial domain adaptation. To overcome the aforementioned difficulties, we propose an innovative DAT approach, namely Max-margin Domain-Adversarial Training (MDAT), to realize stable and comprehensive domain alignment. To demonstrate its effectiveness, we develop an Adversarial Reconstruction Network (ARN) that only utilizes MDAT for UDA. Specifically, ARN consists of a shared feature extractor, a label predictor, and a reconstruction network (i.e. decoder) that serves as a domain classifier. Supervised learning is conducted on source domain, and MDAT helps learn domain-invariant features. In MDAT, the decoder only focuses on reconstructing samples on source domain and pushing the target domain away from a margin, while the feature extractor aims to fool the decoder by learning to reconstruct samples on target domain. In this way, three critical issues can be solved by MDAT: the max-margin loss reduces the discriminative capacity of domain classifier, leading to balanced and thus stable adversarial training; without involving new network structures, MDAT achieves both pixel-level and feature-level domain alignment; visualizing the reconstructed samples reveals how the source and target domains are aligned. We evaluate ARN with MDAT on five visual and non-visual UDA benchmarks. It achieves significant improvement to DAT on all tasks with pixel-level or higher-level domain shift. We also observe that it is insensitive to the choices of hyperparameters and as such is favorable for replication in practice. In principle, our approach is generic and can be used to enhance any UDA methods that leverage domain alignment as an ingredient. Domain adaptation aims to transfer knowledge from one domain to another. provide an upper bound of the test error on the target domain in terms of the source error and the H H-distance. As the source error is stationary for a fixed model, the goal of most UDA methods is to minimize the H H-distance by reducing some metrics such as Maximum Mean Discrepancy (MMD) and CORAL . Inspired by Generative Adversarial Networks (GAN) , proposed to learn domain-invariant features by adversarial training, which has inspired many UDA methods thereafter. Adversarial Discriminative Domain Adaptation (ADDA) tried to fool the label classifier by adversarial training but not in an end-to-end manner. CyCADA and PixelDA leveraged GAN to conduct both feature-level and pixel-level domain adaptation, which yields significant improvement yet the network complexity is high. Another line of approaches that are relevant to our method is the reconstruction network (i.e. the decoder network). The success of image-to-image translation corroborates that it helps learn pixellevel features in an unsupervised manner. employed a decoder network for pixel-level adaptation, and Domain Separate Network (DSN) further leveraged multiple reconstruction networks to learn domain-specific features. These approaches treat the decoder network as an independent component that is irrelevant to domain alignment . In this paper, our approach proposes to utilize the decoder network as domain classifier in MDAT which enables both feature-level and pixel-level domain alignment in a stable and straightforward fashion. In unsupervised domain adaptation, we assume that the model works with a labeled dataset X S and an unlabeled dataset X T. Let X S = {(x s i, y s i)} i∈ [Ns] denote the labeled dataset of N s samples from the source domain, and the certain label y s i belongs to the label space Y that is a finite set (Y = 1, 2, ..., K). The other dataset X T = {x t i} i∈ [Nt] has N t samples from the target domain but has no labels. We further assume that two domains have different distributions, i.e. The proposed architecture is composed of a shared feature extractor G e for two domains, a label predictor G y and a reconstruction network G r. In addition to the basic supervised learning in the source domain, our adversarial reconstruction training enables the extractor G e to learn domain-invariant features. Specifically, the network G r aims to reconstruct the source samples x s and to impede the reconstruction of the target samples x t, while the extractor G e tries to fool the reconstruction network in order to reconstruct the target samples x t. trained to determine whether the input sample belongs to the source or the target domain while the feature extractor learns to deceive the domain classifier, which is formulated as: In DAT, we usually utilize CNN as the feature extractor and fully connected layers (FC) as the domain classifier. DAT reduces the cross-domain discrepancy, achieving significant performance improvement for UDA. Nevertheless, the training of DAT is rather unstable. Without sophisticated tuning of the hyper-parameters, DAT cannot reach the convergence. Through empirical experiments, we observe that such instability is due to the imbalanced minimax game. The binary domain classifier D can easily achieve convergence with very high accuracy at an early training epoch, while it is much harder for the feature extractor F to fool the domain classifier and to simultaneously perform well on the source domain. In this sense, the domain classifier dominates DAT, and the only solution is to palliate the training of D by tuning the hyper-parameters according to different tasks. In our method, we restrict the capacity of the domain classifier so as to form a minimax game in a harmonious manner. Inspired by the max-margin loss in Support Vector Machine (SVM) (i.e. hinge loss), if we push the source domain and the target domain away from a margin rather than as far as possible, then the training task of F to fool D becomes easier. For a binary domain classifier, we define the margin loss as where y is the predicted domain label, [·] +:= max(0, ·), m is a positive margin and t is the ground truth label for two domains (t = −1 for the source domain and t = 1 for the target domain). Then we introduce our MDAT scheme based on an innovative network architecture. Besides the training instability issue, DAT also suffers from restrictive feature-level alignment -lack of pixel-level alignment. To realize stable and comprehensive domain alignment together, we first propose an Adversarial Reconstruction Network (ARN) and then elaborate MDAT. As depicted in Figure 1, our model consists of three parts including a shared feature extractor G e for both domains, a label predictor G y and a reconstruction network G r. Let the feature extractor G e (x; θ e) be a function parameterized by θ e which maps an input sample x to a deep embedding z. Let the label predictor G y (z; θ y) be a task-specific function parameterized by θ y which maps an embedding z to a task-specific predictionŷ. The reconstruction network G r (z; θ r) is a decoding function parameterized by θ r that maps an embedding z to its corresponding reconstructionx. The first learning objective for the feature extractor G e and label predictor G y is to perform well in the source domain. For a supervised K-way classification problem, it is simply achieved by minimizing the negative log-likelihood of the ground truth class for each sample: where y s i is the one-hot encoding of the class label y s i and the logarithm operation is conducted on the softmax predictions of the model. The second objective is to render the feature learning to be domain-invariant. This is motivated by the covariate shift assumption that indicates if the feature distributions S(z) = {G e (x; θ e)|x ∼ D S } and T (z) = {G e (x; θ e)|x ∼ D T } are similar, the source label predictor G y can achieve a similar high accuracy in the target domain. To this end, we design a decoder network G r that serves as a domain classifier, and then MDAT could be applied for stable training. Different from the normal binary domain classifier, MDAT lets the decoder network G r only reconstruct the features in the source domain and push the features in the target domain away from a margin m. In this way, the decoder has the functionality of distinguishing the source domain from the target domain. The objective of training G r is formulated as where m is a positive margin and L r (·) is the mean squared error (MSE) term for the reconstruction loss that is defined as where || · || 2 2 denotes the squared L 2 -norm. Oppositely, to form a minimax game, the feature extractor G e learns to deceive G r such that the learned target features are indistinguishable to the source ones, which is formulated by: Then the whole learning procedure of ARN with MDAT can be formulated by: where L y denotes the negative log-likelihood of the ground truth class for labeled sample (x s i, y s i) and α controls the interaction of the loss terms. In the following section, we provide theoretical justifications on how MDAT reduces the distribution discrepancy, and discuss why it is superior to the classic DAT. In this section, we provide the theoretical justifications on how the proposed method reduces the distribution discrepancy for UDA. The rationale behind domain alignment is motivated from the learning theory of non-conservative domain adaptation problem by Ben-David et al. : Theorem 3.1 Let H be the hypothesis space where h ∈ H. Let (D S, s) and (D T, t) be the two domains and their corresponding generalization error functions. The expected error for the target domain is upper bounded by where Theoretically, when we minimize the H H-distance, the upper bound of the expected error for the target domain is reduced accordingly. As derived in DAT , assuming a family of domain classifiers H d to be rich enough to contain the symmetric difference hypothesis set of H p, such that H p H p = {h|h = h 1 ⊕ h 2, h 1, h 2 ∈ H p} where ⊕ is XOR-function, the empirical H p H p -distance has an upper bound with regard to the optimal domain classifier h: whereD S andD T denote the distributions of the source and target feature space Z S and Z T, respectively. Note that the MSE of G r plus a ceiling function is a form of domain classifier h(z), i.e. [m − L r (·)] + − 0.5 for m = 1. It maps source samples to 0 and target samples to 1 which is exactly the upper bound in Eq.10. Therefore, our reconstruction network G r maximizes the domain discrepancy with a margin and the feature extractor learns to minimize it oppositely. Compared with the conventional DAT-based methods that are usually based on a binary logistic network , the proposed ARN with MDAT is more attractive and incorporates new merits conceptually and theoretically: Stable training and insensitivity to hyper-parameters. Using the decoder as domain classifier with a margin loss to restrain its overwhelming capacity in adversarial training, the minimax game can continuously provide effective gradients for training the feature extractor. Moreover, through the experiments in Section 4, we discover that our method shows strong robustness to the hyperparameters, i.e. α and m, greatly alleviating the parameters tuning for model selection. Richer information for comprehensive domain alignment. Rather than DAT that uses a bit of domain information, MDAT utilizes the reconstruction network as the domain classifier that could capture more domain-specific and pixel-level features during the unsupervised reconstruction . Therefore, MDAT further helps address pixel-level domain shift apart from the feature-level shift, leading to comprehensive domain alignment in a straightforward manner. Feature visualization for method validation. Another key merit of MDAT is that MDAT allows us to visualize the features directly by the reconstruction network. It is crucial to understand to what extent the features are aligned since this helps to reveal the underlying mechanism of adversarial domain adaptation. We will detail the interpretability of these adapted features in Section 4.3. In this section, we evaluate the proposed ARN with MDAT on a number of visual and non-visual UDA tasks with varying degrees of domain shift. We conduct ablation study to corroborate the effectiveness of MDAT and unsupervised reconstruction for UDA. Then the sensitivity of the hyperparameters is investigated, and the adapted features are interpreted via the reconstruction network in ARN. Setup. We evaluate our method on four classic visual UDA datasets and a WiFi-based Gesture Recognition (WGR) dataset . The classic datasets have middle level of domain shift including MNIST , USPS , Street View House Numbers (SVHN) and Synthetic Digits (SYN). For a fair comparison, we follow the same CNN architecture as DANN while using the inverse of G e as G r with pooling operation replaced by upsampling. For the penalty term α, we choose 0.02 by searching over the grid {10 −2, 1}. We also obtain the optimal margin m = 5 by a search over {10 −1, 10}. Then we use the same hyperparameter settings for all tasks to show the robustness. For the optimization, we simply use Adam Optimizer (lr = 2 × 10 −4, β 1 = 0.5, β 2 = 0.999) and train all experiments for 50 epochs with batch size 128. We implemented our model and conducted all the experiments using the PyTorch framework. More implementation details are illustrated in the appendix. Baselines. We evaluate the efficacy of our approach by comparing it with existing UDA methods that perform three ways of domain alignment. Specifically, MMD regularization and Correlation Alignment employ the statistical distribution matching. DRCN and DSN Table 1: We compare with general, statistics-based (S), reconstruction-based (R) and adversarialbased (A) state-of-the-art approaches. We repeated each experiment for 3 times and report the average and standard deviation (std) of the test accuracy in the target domain. while many prevailing UDA methods adopt domain-adversarial training including DANN , ADDA , MECA , CyCADA and CADA . For all transfer tasks, we follow the same protocol as DANN that uses official training data split in both domains for training and evaluates the testing data split in the target domain. Both datasets are composed of grey-scale handwritten images with diverse stroke weights, leading to low-level domain shift. Since USPS has only 7291 training images, USPS→MNIST is more difficult. As shown in Table 1, our method achieves state-of-the-art accuracy of 98.6% on MNIST→USPS and 98.4% on USPS→MNIST, which demonstrates that ARN can tackle low-level domain shift by only using ART (rather than many adversarial UDA methods that adopt other loss terms to adjust classifier boundaries or conduct style transfer). SVHN→MNIST and SYN→SVHN. The SVHN dataset contains RGB digit images that introduce significant variations such as scale, , embossing, rotation, slanting and even multiple digits. The SYN data consists of 50k RGB images of varying color, , blur and orientation. These two tasks have tremendous pixel-level domain shfit. The proposed method achieves a state-ofthe-art performance of 97.4% for SVHN→MNIST, far ahead of other DAT-based methods, significantly improving the classic DANN by 22.7%. Similarly, ARN with MDAT also achieves a noticeable improvement of 5.3% compared with the source-only model, even outperforming the supervised SVHN accuracy 91.3%. WiFi Gesture Recognition with Distant Domains. To evaluate the proposed method on a non-visual UDA task, we applied our method to the WiFi gesture recognition dataset . The WiFi data of six gestures was collected in two rooms regarded as two domains. The in Table 2 demonstrate that our approach significantly improves classification accuracy against Source-Only and DANN by 32.9% and 23.1%, respectively. Table 3: The accuracy (%) with different hyperparameters on SVHN→MNIST. The contribution of MDAT and image reconstruction in ARN. We design an ablation study to verify the contribution of MDAT and unsupervised reconstruction in ARN. To this end, we discard the term L r (x t) in Eq.4, and evaluate the method, denoted as ARN w.o. MDAT in Table 1. Comparing ARN w.o. MDAT with source-only model, we can infer the effect of unsupervised reconstruction for UDA. It is observed that ARN w.o. MDAT improves tasks with low-level domain shift such as MNIST↔USPS, which conforms with our discussion that the unsupervised reconstruction is instrumental in learning low-level features. Comparing ARN w.o. MDAT with the original ARN, we can infer the contribution of MDAT. Table 1 shows that the MDAT achieves an impressive marginof-improvement. For USPS→MNIST and SVHN→MNIST, the MDAT improves ARN w.o. MDAT by around 30%. It demonstrates that MDAT which helps learn domain-invariant representations is the main reason for the tremendous improvement. Parameter sensitivity. We investigate the effect of α and m on SVHN→MNIST. The in Table 3 show that ARN achieves good performance as α ∈ [0.01, 0.1] and even with larger α ARN is able to achieve convergence. In comparison, denoting α as the weight of adversarial loss, the DANN cannot converge when α > 0.2. For the sensitivity of m, the accuracy of ARN exceeds 96.0% as m ≥ 1. These analyses validate that the training of ARN is not sensitive to the parameters and even in the worst cases ARN can achieve convergence. Gradients and training procedure. We draw the training procedure with regard to loss and target accuracy in Figure 2 (b) and Figure 2(a), respectively. In Figure 2 (b), ARN has smoother and more effective gradients (L r) for all α, while the loss of DAT domain classifier (L d) gets extremely small at the beginning. This observation conforms with our intuition, which demonstrates that by restricting the capacity of domain classifier MDAT provides more effective gradients for training feature extractor, leading to a more stable training procedure. This could be further validated in Figure 2 (b) where the ARN accuracy is more stable than that of DAT across training epochs. Table 4: Visualizing the source image, target images and reconstructed target images (R-Target Images) for four digit adaptation tasks. Interpreting MDAT features via reconstructed images. One of the key advantages of ARN is that by visualizing the reconstructed target images we can infer how the features are domain-invariant. We reconstruct the MDAT features of the test data and visualize them in Table 4. It is observed that the target features are reconstructed to source-like images by the decoder G r. As discussed before, intuitively, MDAT forces the target features to mimic the source features, which conforms with our visualization. Similar to image-to-image translation, this indicates that our method conducts implicit feature-to-feature translation that transfers the target features to source-like features, and hence the features become domain-invariant. We analyze the performance of domain alignment for DANN (DAT) and ARN (MDAT) by plotting T-SNE embeddings of the features z on the task SVHN→MNIST. In Figure 3 (a), the source-only model obtains diverse embeddings for each category but the domains are not aligned. In Figure 3 (b), the DANN aligns two domains but the decision boundaries of the classifier are vague. In Figure 3 (c), the proposed ARN effectively aligns two domains for all categories and the classifier boundaries are much clearer. We proposed a new domain alignment approach namely max-margin domain-adversarial training (MDAT) and a MDAT-based network for unsupervised domain adaptation. The proposed method offers effective and stable gradients for the feature learning via an adversarial game between the feature extractor and the reconstruction network. The theoretical analysis provides justifications on how it minimizes the distribution discrepancy. Extensive experiments demonstrate the effectiveness of our method and we further interpret the features by visualization that conforms with our insight. Potential evaluation on semi-supervised learning constitutes our future work. Hyperparameter For all tasks, we simply use the same hyperparameters that are chosen from the sensitivity analysis. We use α = 0.02 and m = 5.0, and we reckon that better can be obtained by tuning the hyperparameters for specific tasks. Network Architecture For a fair comparison, we follow the network in DANN for digit adaptation and simply build the reconstruction network by the inverse network of the extractor. Here we draw the network architectures in Table 5. For WiFi gesture recognition, we adopt the same architecture as CADA that is a modified version of LeNet-5. We have presented all the of the sensitivity study in Section 4.2, and now we show their detailed training procedures in Figure 4 (a) and 4(b). It is observed that the accuracy increases when α drops or the margin m increases. The reason is very simple: when α is too large, it affects the effect of supervised training on source domain; when the margin m is small, the divergence between source and target domain (i.e. H H-distance) cannot be measured well. Here we provide more visualization of the reconstructed images of target samples. In Figure 5, the target samples are shown in the left column while their corresponding reconstructed samples are shown in the right. We can see that for low-level domain shift such as MNIST↔USPS, the reconstructed target samples are very source-like while preserving their original shapes and skeletons. However, for larger domain shift in Figure 5 (c) and 5(d), they are reconstructed to source-like same digits but simultaneously some noises are removed. Specifically, in Figure 5 (d), we can see that one target sample (SVHN) may contain more than one digits that are noises for recognition. After reconstruction, only the right digits are reconstructed. Some target samples may suffer from terrible illumination conditions but their reconstructed digits are very clear, which is amazing.

Represent the short summary for finding the corresponding scientific article (long)

A stable domain-adversarial training approach for robust and comprehensive domain adaptation

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Pseudo-labeling has shown to be a weak alternative for semi-supervised learning. We, conversely, demonstrate that dealing with confirmation bias with several regularizations makes pseudo-labeling a suitable approach.


Question: Represent the scientific article to find a short TLDR

Question: We present an approach for expanding taxonomies with synonyms, or aliases. We target large shopping taxonomies, with thousands of nodes. A comprehensive set of entity aliases is an important component of identifying entities in unstructured text such as product reviews or search queries. Our method consists of two stages: we generate synonym candidates from WordNet and shopping search queries, then use a binary classiﬁer to ﬁlter candidates. We process taxonomies with thousands of synonyms in order to generate over 90,000 synonyms. We show that using the taxonomy to derive contextual features improves classiﬁcation performance over using features from the target node alone. We show that our approach has potential for transfer learning between diﬀerent taxonomy domains, which reduces the need to collect training data for new taxonomies. Semantic Networks (SN) represent entities, relationships between entities, and their properties. Semantic Networks may represent a broad variety of information, from named entities, such as persons or places, to abstract concepts. The term "knowledge graph" is also used to describe this form of structured data. One of the properties commonly encoded in a SN are the primary name and aliases of an entity in multiple languages. For example, Wikidata 1 entity Q2 has multilingual names, such as Earth or Blue Planet (in English), or Tierra (in Spanish). Semantic networks may include sub-structures based on a subset of the relations defined, for example, taxonomies which define type-subtype relations; for example, ConceptNet includes the WordNet taxonomy BID28 as a subset of its nodes and relations. Synonyms, or aliases, are equivalent names for entities in a SN. For example, "washing machine" and "washer" can refer to the same concept of an appliance type. Synonyms enable improved performance in a variety of SN applications. For entity extraction from text BID7 , ], wikification BID18 BID5, or natural language instruction grounding BID16, a broader set of synonyms improves recall. In applications which use SN to generate prompts for users, such as conversational agents BID11 BID31 or generating explanations of the system's state in natural language, a richer set of synonyms in more varied utterances. In this paper, we focus on the problem of expanding taxonomies with synonyms for applications in which entities are complex concepts arranged into taxonomies designed to facilitate browsing the product catalog on amazon.com. The ontologies contain product type taxonomies, which are the focus for this work, in addition to other information such as attributes for refining products in search . In addition to distinct product types, the taxonomies contain nodes which are complex concepts, for example combinations of types and attributes, or groupings of multiple types. For example, the node "Gloves & Protective Gear" groups together gloves and other gear; the node "Automatic Irrigation Equipment" describes irrigation equipment that has automation features. The primary application of the synonyms generated using our method is to identify direct references to the taxonomy nodes in text such as search queries. Having a broader set of synonyms for taxonomy nodes enables a broader query coverage for experiences that are specific to products in the taxonomy, for example, showing the best selling products under a given category. It is thus important to the users' experience that node synonyms are as accurate as possible, within the broader context of the taxonomy. For example, given the node "household bathroom surface cleaners" we output synonyms such as "family bathroom surface cleaner" and "house bath surface cleansing." Our method is robust to errors of word sense compatibility, for example we reject "mack game restrainer" as a synonym for "mac game controllers," or "store circuit board" is a rejected candidate for "memory cards."The taxonomies are authored by experts familiar with the respective shopping domains to facilitate navigation and browsing (Section 4.1). They contain over 4,300 nodes and have depths of over 30 nodes; in addition to taxonomical relationships, they represent type properties, possible values, node equivalence, and other information. In this paper, we identify each taxonomy by its root node name. For the example shown in Figure 1, the taxonomy "Baby Products" includes, among 15 other nodes, a category node named "Car Seats and Accessories. " This has the children "Car Seats," "Car Seat Bases," "Car Beds," and "Accessories. " The "Accesories" node has 17 children (e.g. "Cup Holders" and "Seat Liners"), while the "Car Seats" node has five children grouped by age group and chair type. We note the fine granularity of nodes, which includes distinctions based on product types, features, indented use, and other criteria dependent on the domain; concepts range from general to specific in fine increments, with children refining and specifying the parent node. The taxonomy nodes we target have complex names, for example "Convertible Child Safety Car Seats" and are thus unlikely to be frequently found in large natural language text corpora with sufficient frequency in order to extract synonyms from unstructured text. We present a method that leverages similarity within the taxonomy to evaluate synonym candidates obtained using low-precision, high-recall methods. Our goal is to enable collecting possible synonyms from a broad range of sources, and output a final set of synonyms consistent to a single standard. This method enables expansion with synonyms for complex SN that are not common in typical text corpora, such as shopping taxonomies for browsing. The main advantages of our approach are that: 1) it does not depend on frequent mentions in corpora of entities in the taxonomy; 2) it identifies synonyms that fit within the broader structure of a taxonomy contained within the graph, and outputs synonyms of similar specificity to the original name; 3) the classifier uses domain-independent features, enabling cross-domain predictions. Our method consists of the following stages (Figure 2):1. Generate synonym candidates for each node of the taxonomy. We experimented with two methods of candidate generation. First, we primarily used a method based on Figure 1: Sample section of a taxonomy used in this work, which is designed for exploring and filtering online shopping catalogs. We highlight the path from the root node, "Baby Products," to a leaf node, "Child Safety Booster Car Seats. " Each node prefixed by a + sign indicates the node has children; leaf nodes are marked by a -. For compactness, we enumerate instead of indenting some of the 15 children of the root node. Figure 2: Overview of our method. We start with product taxonomies designed for browsing a large online shopping catalog, described in Section 4.1, and generate synonym candidates for each node using a thesaurus such as WordNet (Section 3.1). We then classify the set of candidates using a binary classifier (Section 3.2) to output the final set of synonyms. WordNet BID23, to generate the cartesian product of concept-level synonyms that are present in the node's name (Section 3.1). Secondly, we show additional on classifying shopping search queries (Section 4.4).2. Filter synonym candidates using a binary classifier (Section 3.2). The classifier uses features derived from a) similarity between the candidate the target node, and b) similarity features between the candidate and other nodes in the taxonomy. Our goal is to avoid producing synonyms more general or more specific than the original node name, such that the synonyms are consistent with the taxonomy as a whole. The classifier uses features independent of the taxonomy vocabulary, making our method suitable for transfer learning by predicting on new taxonomies that do not have training data available. Transfer learning is one method of interest to reduce the need to collect training labels for new taxonomies. The rest of the paper is structured as follows. We first review relevant literature. We then describe the taxonomies we use in this work (Section 4.1), and the methods of obtaining synonym candidates and classifying them. We then evaluate the binary synonym classifier using a corpus of annotations collected using crowdsourcing for synonyms generated using the thesaurus. We also include cross-domain learning experiments to evaluate the potential for training the classifier on one taxonomy and predicting on synonyms for different taxonomy (Section 4.3). Furthermore, we conducted a separate evaluation using an alternative method of selecting synonym candidates, which we will briefly summarize: we associated search queries with taxonomy names using customer purchases, and used these search terms as synonym candidates (Section 4.4). We evaluate the impact of using domain-specific knowledge, specifically lists of known brand names, which may be closely associated but not synonymous with product categories, to improve synonym filtering. We conclude the paper with observations about the role of taxonomy-wide similarity in predicting synonymy and describe future directions. Methods of automatically buiding knowledge bases from unstructured text include identifying salient terms (keywords and keyphrases), identifying synonymy between terms, forming entities and entity hierarchies from keywords, and inferring relationships and rules between entities BID4; this covers the spectrum from keyword extraction to semantic inference. Synonym extraction has been defined as one of the base steps of automated ontology and knowledge base building. Synonym expansion has been used to enrich search keywords in order to improve recall. WordNet synonym sets have been used to expand search keywords BID29. This work used rules on the WordNet taxonomy to expand queries with keywords, increasing the number of words that are matched. Other work has used text mapping to concepts in a thesaurus for query expansion BID1. In life sciences, domain-specific ontologies have been used in similar ways to expand search terms BID33.Previous work in extracting synonyms from text relies on frequent mentions of the entities in text corpora BID6 BID19 BID32 BID15. These works identify synonyms based on statistically similar contexts or phrases in which they are used. The taxonomies we use in this work are engineered to facilitate exploring a product catalog for online shopping, for example by choosing subtypes of products or filtering by feature. As such, we cannot expect that all node names and their potential synonyms will occur in text. Other work has used clustering methods that identify synonyms such as "birth date" and "date of birth" in search queries BID14. Topic-based methods have also been used in medical document search BID17 to add aliases as encountered in text. We show an approach that uses search queries and past customer purchases to propose synonym candidates (Section 4.4); this method benefits from using domain-specific knowledge, such as product brand names. Other work in identifying aliases has been focused on named entities, as opposed to common words. Named entity recognition and extraction (NER) are rich research areas, focusing on identifying and categorizing proper names in text BID8 BID25 BID24 BID13. Named entities may refer to persons, places, brands, or authors. It is a problem related to synonym detection since the same person may be referred to in different contexts using different names. The problem we address in this paper is different than existing work in NER and synonym extraction from unstructured text. Our entities cannot be considered names entities, and as mentioned above, we do not expect to find them mentioned frequently in text corpora. Structure mapping is an established approach of comparing semantically-complex structures, with applications in analogy modeling BID12 BID20 BID9 BID10. Structural similarity methods consider correspondence at a relational level to be indicative of higher similarity, as opposed to feature-based similarity models that compare attributes directly. For example, in a structural similarity approach, the function an entity performs is more important than individual features such a color. Previous work has used structural similarity in contexts derived from robot tasks in order to identify equivalent objects for a given task. Other work using structural similarity for comparison identified significant discrepancies in superficially similar structures, for example to make the distinction between an arch and a bridge BID21. In designing the synonym filtering classifier, we incorporated concepts from structure mapping by adding features that compare the synonym candidate with multiple nodes in the taxonomy; we refer to these as structural similarity features. The rationale behind this decision was to calibrate the level of generality of a synonym by considering the surrounding nodes of the taxonomy. Our approach consists of two stages. First, we identify synonym candidates. We describe a method based on a thesaurus containing concepts part of a node's primary name (Section 3.1). Second, we filter synonym candidates using a classifier (Section 3.2). The rationale behind our design is that multiple methods can be used to generate candidates; we demonstrate a generally-applicable method. The filtering stage processes candidates regardless of the method used to obtain them. We start from the observation that many taxonomy node names are composed of common words for which we could identify synonyms. For example, the node "Baby Clothing & Shoes" has the synonym "Baby Clothing & Footwear." One of the challenges in this approach is selecting word-level synonyms that are consistent with the sense of the original word. We identify word synonyms using WordNet BID23. WordNet represents concepts as synonym sets, or synsets, arranged in a taxonomy. Each synset consists of synonyms for a concept corresponding to a particular sense of the word. For example the word "car" may have the meanings Car.n.01: car, auto, automobile; Car.n.02: car, rail car, railroad car; Car.n.03: car, gondola. For each node in the taxonomies:1. We split the node's name into separate concepts found in WordNet. We combine individual consecutive words into concepts whenever possible. 2. We select all synsets for the words in the node name. 3. We perform word sense disambiguation in order to find the WordNet synset to the taxonomy. We select the WordNet synset most similar to the node and its context for each concept identified in the node name. We define context as the node name and the other node names in the taxonomy. We compute similarity by averaging cosine similarity over word embeddings between all pairwise concept pairs between the synonym set and the node context. We choose the synset that in the highest average similarity to use for permuting word synonyms. The majority of words in our taxonomies have more than one synset in WordNet, we designed this step to prune the set of incorrect candidates. 4. We generate the cartesian product of all synset words extracted from WordNet for each concept in the node's name. The following is an example:1. Given a node name "washing machine parts," we identify the corresponding concepts, "washing machine" and "parts." 2. We build a context vocabulary by sampling other node names from the Electronics taxonomy. We compare all synsets corresponding to "washing machine" and "parts" to the context vocabulary. We select the most similar sense for each concept, "washer.n.03" and "part.n.01." 3. We extract all lemmas for each synset, i.e. "washer," "automatic washer" and "washing machine," and "part," "portion," "component part," "component." 4. We generate the cartesian product of these lemmas starting from the original node name, for example "washer parts" or "automatic washer components," for a total of 12 phrases. Each ing list of lemmas is a synonym candidate, which is then accepted or rejected using the method described below. We select candidates that 1) have similar meaning to the original node name, and 2) have a similar level of generality with the original name node, in the context of the taxonomy. In common speech, the product types "TV" and "LED TV" may be considered equivalent, with most shoppers referring to an LED TV as simply TV, assuming that LED is the most common display technology at a given time. However, from our taxonomy standpoint they are not synonyms: the type TV has other sub-types, such as OLED TV, Plasma TV, CRT TV, and LED TV. We hypothesized that, in order to make such distinctions when identifying synonyms, taking into account the structure of the taxonomy is an essential factor. We use vector word representations, or word embeddings, to compute some of the classifier's features by comparing synonym candidates with various node names in the taxonomy. To compute features, we sum the corresponding vectors for each word in a taxonomy node name or the synonym candidate(using bi-and tri-grams when available); then, we compute the cosine similarity value between the ing addition vectors. We used Numberbatch, a set of word embeddings generated the ConceptNet semantic network, because it is publicly available, it includes WordNet concepts, and has low bias BID28. We implemented a binary gradient boosting classifier using the Python scikit-learn tookit BID26.Many entity names and synonym candidates consist of more than one word, and have corresponding embeddings, such as washing machine. We identify multi-word concepts by searching WordNet with adjacent n-grams in decreasing order of length, in a greedy approach that extracts the longest match from the node name. Synonym candidates generated using WordNet similarly may consist of multi-word concepts as synonyms for single words. We compute the word embedding distances between node names and synonym candidates by first summing all respective vectors for the node and the synonym candidate, then computing cosine similarity between the ing vectors. We group features in local and structural, where local features are computed only with respect to the target node, and structural features refer to other nodes in the taxonomy.• Local Features: -Word frequency in search queries: We compute four features, for the average and minimum frequency of words in the original node name and in the synonym string. We denote this group of features W F;-Character and word edit distance: We compute two features from edit distance in character and in words between the candidate and the original name (edit(SC, N));-Cosine similarity: We compute one feature by summing the word embedding vectors of the synonym candidate words (SC), and comparing this sum with the word embedding sum of the original node name (d (SC, N) ).• Structural features: We compute word embedding similarity between the synonym candidate and the following, and add one feature for each (examples refer to nodes shown in Figure 1):-The node's parent name, e.g. "Child Safety Car Seats & Accessories" for "Car Seats" (d(SC, P))-The name of the taxonomy root, "Baby Products" (d (SC, R) ).-The average distance to the node's direct children (d(SC, ci), where ci ∈ N.children), if the node has children, e.g. all nodes including "Infant Safety Car Seats," "Forward Facing Child Safety Car Seats," and "Baby Stroller Travel Systems."Our hypothesis, which we test in Section 4.3 using feature ablation, is that comparing a synonym candidate with the broader structure of the taxonomy models how specific a node is with respect to others. We first describe the taxonomies we used in the evaluation (Section 4.1). We then describe the methodology for collecting training data for training classifiers and show for classifying synonyms generated using WordNet, exploring feature contribution and transfer learning between taxonomies (Sections 4.2 and 4.3). We include separate of predicting synonyms from search queries and the effect of using domain-specific features (Section 4.4). We use taxonomies created by manual ontology design and aimed to enable navigation through the amazon.com product catalog. The taxonomies enable users to refine products by category and features, for example by selecting a product type after performing a search. We used the following eight taxonomies, with total number of nodes shown in parentheses, which we selected to cover a broad range of product types: The taxonomies contain type nodes on the leaves, for example "Bulb Planters," and grouping nodes at higher levels, for example "Gardening & Lawn Care" is a grouping node with children such as "Hand Tools," which in turn has children such as "Picks," "Bulb Planters," or "Manual Lawn Aerators." FIG1 shows a histogram of the number of words in node names in our taxonomies; for example, the node name "women's contemporary clothes designer base layer sets" has seven words. The majority of nodes have three or more words in the name. The taxonomies contain 2379 distinct words. DISPLAYFORM0 We generated candidates using the WordNet method described in Section 3.1. For the total of 15,197 nodes in the eight taxonomies, we generated 182,974 synonym candidates. We took a uniform sample, and labeled them using a crowdsourced survey 2. We sampled uniformly 3331 nodes with distinct names and a total of 4488 corresponding synonym candidates for the eight taxonomies. The candidate synonym set contains 3056 distinct words (0.91 distinct words per node). We designed the survey to present the original node name, a synonym candidate, and the category name. We chose the context as the name of the root in each taxonomy tree, for example "Electronics." Participants were asked to answer a binary yes/no question, "Do these phrases mean the same in the following context?" We collected 10 answers from separate participants for each name-synonym candidate pair. We calculated the proportion of "yes" answers and used a threshold, which we refer to as the annotation threshold, to assign the synonym example a final label, positive or negative. We used all eight taxonomies and collected ten answers per question, totaling 45,000 responses. FIG2 shows the proportion of positive answers per evaluation. We observe a skew towards positive answers, which is to be expected given that the generation method incorporates word sense disambiguation to reduce the number of implausible candidates. We use this label set in Section 4.3, and explore the effect of choosing an annotation threshold for this label set. We trained the classifier on 90% of the labeled data and tested it on the remaining 10%. We supplemented each instance in the training set with ten negative examples by selecting names of other nodes in the taxonomy; for example, in the Electronics taxonomy, we provided examples such as "mp3 player" and "televisions" as a negative synonym example for "LED TVs." We trained and tested using this process 50 times for each condition, and report average precision and recall values. We explored the effect of the annotator agreement threshold at which the annotation is considered positive; for this set of experiments we enabled all features of the classifier. TAB0 shows classification performance for different annotation thresholds and train/test splits in the Electronics taxonomy. For the remained of the experiments using crowdsourced labels, we select an annotation threshold of 0.6, as it achieved high precision (our primary consideration) and satisfactory recall. We observed that using a combination of local and structural features in the best performance (0.92 F-1 score), compared to local features or structural features alone Table 2: Classification performance (precision, P, recall, R, and F-1 score) for the Electronics taxonomy for classifiers using subsets of the features described in Section 3.2. We observe that including some structural features, by comparing synonym candidates with the root, parent or children of the target node, improves classification accuracy. DISPLAYFORM0 (0.56 and 0.79 respective F-1 scores). We conducted an ablation study to investigate the contribution of each category of feature. We disabled different combinations of features and trained separate classifiers for each subset of features. For brevity, we report representative findings in the Electronics category (Table 2). We note that in isolation, either local features (row 2) or structural features (row 3) in lower recall and precision, with combinations of the two ing in improved performance. The cosine similarity between the candidate between the target name and the candidate is important: removing it decreases recall to 67% (row 7), and on its own this feature achieves an F-1 score of 0.77 (row 11). Including a mix of local and structural features yields in similar performance of 94% precision and a 89-90% recall (rows 4, 5, 8, 10). We observe lower F-1 scores when using all three structural features, and improvements when selecting two out the three. Frequency-based features have high precision but low recall (row 12), which is expected since, if the keywords are present in large volume in search queries, it is likely that they are meaningful phrases. The thesaurusbased method generates invalid candidates primarily because of the individual word senses do not have the same meaning when put together, which means that the combination will have lower search term frequency. However, it does not identify less frequent terms, which is where in practice synonyms are most useful for, since they allow interpreting infrequent queries. We observed similar performance in the other taxonomies, with F-1 scores of over 0.83 (with the exception of the "Clothing, Shoes & Jewelry). We selected the feature set on row 4 in Table 2, i.e. all features except for d(SC, R), and evaluated prediction performance in all other categories. Table 3 shows the best performance for each of the eight product type taxonomies. We note similar scores, with the exception of Clothing, which had a significant Table 3: Classification for each product taxonomy, using 10% of annotations for testing; showing average values for 50 train/test samples. We used the feature set shown on row 4 in Table 2.skew towards positive annotations. Including the candidate-root similarity features ed in lower performance, similar to our previous observation (average F-1 score of 0.78, lower than the average of 0.86 shown in Table 3). Finally, we conducted cross-domain experiments and observed degraded but comparable classification performance to using the same taxonomy for training and testing. We trained a classifier using one taxonomy and used it to predict in another. This is possible because none of the features described in this section rely on domain-specific information such as known word-level labels. We experimented with all 56 combinations of source-target combinations. For this set of experiments, we used the full feature set (i.e. not excluding d(SC, R), since it ed in a higher average F-1 score. In Table 4 we show , in order of F-1 score, of the top 5 best performing and top 5 worst performing from all pairwise combinations of taxonomies; for the sake of brevity we exclude the full list. The average F-1 score over all 56 combinations was 0.766 with a standard deviation of 0.087. We observe that for all top five worst performing, the target taxonomy is Clothing, Shoes & Jewelry; this was the taxonomy with the lowest score in our in-domain experiments (Table 2). The low F-1 scores are due to recall (0.35), precision is competitive (0.94). Similarly, for the top performing target taxonomies Electronics is the most common: precision and recall are both similar but slightly lower on average (0.90 precision compared to 0.94 in-domain, and 0.84 recall compared to 0.90). The prediction from Electronics to Clothing, not shown in the table, has an F-1 score of 0.67. We attribute these changes in performance to label noise, but also to how diverse the taxonomies are. In this section we describe additional using a separate source for synonym candidates. We experimented with selecting synonym candidates from search queries on amazon.com. These associations are derived using product purchases that follow search queries, and using the taxonomy associations for these products BID27. We reference a related method, also applied to an online shopping domain, of associating product attribute values indexed in a shopping catalog BID30. We use user behavior such as Table 4: Classification performance for cross-domain performance, in which we trained on one taxonomy and predicted on another. The table shows the lowest and highest five pairs of source and target taxonomy, ranked by F-1 score. We used the feature set shown on row 1 in Table 2; the full set of features ed in the highest average F-1 performance.product purchases after issuing queries to infer statistical associations between a query and the product; we then use the product's assignment to taxonomy nodes to infer associations between the query and the node. This model associates unique search keyword queries to taxonomy nodes, and outputs a probability distribution over the taxonomy. We selected as synonym candidates queries that occurred frequently, at least once every day for an entire month, and that had a probability of over 80% to lead to purchases from the respective taxonomy node. The following are examples of node names in the Electronics taxonomy, followed by a sample of queries selected for synonym candidates; we selected examples that are not covered by the thesaurus method:• Portable cell phone power banks: "battery bank" • Cell phone cases: "iphone 4s case," "lg g4 phone case" • Repeaters: "wifi repeater" • Hdmi cables: "hdmi to mini displayport cable" While this method has the advantage of accessing a broad and current vocabulary, we also observe a mix of brands, product models, and manufacturer names in those categories; for the task of generating synonyms, these examples are undesirable since they are specific to a subset of items under the taxonomy node, and not to the node globally. This method has the potential of identifying candidates that are not reformulations of the taxonomy nodes. For example, the taxonomy node "Self-Balancing Scooters" may be referred to as "hover boards" in search queries. Using information from search queries enables us to identify emerging synonyms before this information is included in a thesaurus such as WordNet. The queries selected using this method have high lexical variability as a of the variety of intents of search queries. For example, queries for a specific product that is assigned to a single taxonomy node with in a high probability of the query targeting the corresponding node, because the query is unlikely to lead to clicks or purchases other than for the target product. This makes the input vocabulary to the classifier more varied, both in unique words and in the generality of those words; some candidates contain tokens such as model numbers and brands. Using the same methodology as described in Section 4.2, we collected annotations for a uniform sample of 2000 candidate search queries for 337 nodes in the Electronics taxonomy, also with 10 answers per candidate (20,000 answers in total). The 2000 synonym candidates contain 1307 unique words. The ratio of unique candidate words per node is 3.87, 300% higher than the ratio of unique words in original node names per node, of 1.27. The synonym candidate vocabulary is significantly more diverse than for WordNet-generated candidates: for 594 nodes use for collection in Electronics in Section 4.2, there were 1.41 distinct words per node for synonym candidates, only 20% higher than the ratio of 1.17 distinct words in original names per node. Similar to the WordNet survey, we observed a skew towards positive answers. In addition, these annotations showed more positive responses for popular brands in a given product category. We evaluate using domain-specific features, such as known brand names. For candidates that included names of manufacturers or brands, we observed a higher proportion of positive answers for brands that are more common in their respective product category, similar to genericized brands. To control for this effect, we collected an additional set of annotations, authored by experts familiar with product search. For the experiments in this section, we used a set of unigram and bigram embedding vectors generated by applying the word2vec algorithm on the search query dataset BID22. We used the this set of embeddings as they represent brand names in relation to a rich vocabulary, which we evaluate towards the end of this section. Following the same methodology as in Section 4.3 we trained and tested separate classifiers using the crowdsourced or the expert annotations, using all available features. We included a feature in the model, β, that activates when a brand, product line, or manufacturer, is present in the synonym candidate. The list of brands is domain-specific, for example "apple" would be a brand name in Electronics but not in Grocery. We computed β using a subset of 120 brands and manufacturer names extracted from the product catalog. Table 5 shows the effect of the annotation consensus threshold and of using the feature β using the crowdsourced annotations and when training on the expert-annotated candidates. In all these experiments, we used the full feature set described in Section 3.2 and the same setup as in the previous section. Overall, we observe a decline in recall compared to classifying performance on generated synonyms, which we attribute to the greater lexical variety in the valid candidates set; precision is maintained or improved upon compared to candidates generated with WordNet. Classification performance is lower for the expert annotations than for the crowdsourced annotations. Furthermore, using β is beneficial only for the expert annotation set. We attribute these observations to the effect of genericized brands. Annotations diverge between Table 5: Classification performance when training with crowdsourced labels, for the Electronics taxonomy for candidates selected from search query keywords. We evaluated setting an annotation consensus threshold and a domain-specific feature, β, which identifies the presence of brand names in the candidate. Last two rows show classification performance when using expert annotations.crowdsourced and expert annotations for categories in which popular brands are closely identified with the product category. Word embedding vectors trained without domain supervision place those words close to common words denoting the type, since they occur in the same context, making the problem inseparable for the classifier. Introducing domain knowledge, in the form of the known brand feature β, is useful only if the annotation set is free of this conflation between brand and type. Entity aliases are an important component of ontology construction, enabling entity recognition in text and generating natural language references to entities. We demonstrate a method for identifying synonyms for large taxonomies used for online shopping. Our method consists of two complementary approaches of selecting synonym candidates, and a candidate filtering stage which uses a classifier that includes structural similarity features. We show that using structural similarity features, such as comparing synonym candidates with the parent, children, or root nodes in the taxonomy, improves classification accuracy, and that the method is applicable to transfer learning between taxonomies. We include an additional evaluation on using search queries associated statistically with the taxonomy nodes via user behavior. This method extracts a broader vocabulary for the candidates, including tokens that are not common words, such as proper names, model numbers, or years. We show that using domain knowledge such as brand name definitions improves classification performance for candidates extracted from search queries, which conflate in the same context types, brands and other terms. In future work we will experiment with taxonomies in languages other than English. We will explore the potential for predicting synonyms in other languages than the training language, similar to the experiments we showed for cross-domain prediction.

We use machine learning to generate synonyms for large shopping taxonomies.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.