Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Generative Adversarial Networks (GANs) are a very powerful framework for generative modeling. However, they are often hard to train, and learning of GANs often becomes unstable. Wasserstein GAN (WGAN) is a promising framework to deal with the instability problem as it has a good convergence property. One drawback of the WGAN is that it evaluates the Wasserstein distance in the dual domain, which requires some approximation, so that it may fail to optimize the true Wasserstein distance. In this paper, we propose evaluating the exact empirical optimal transport cost efficiently in the primal domain and performing gradient descent with respect to its derivative to train the generator network. Experiments on the MNIST dataset show that our method is significantly stable to converge, and achieves the lowest Wasserstein distance among the WGAN variants at the cost of some sharpness of generated images. Experiments on the 8-Gaussian toy dataset show that better gradients for the generator are obtained in our method. In addition, the proposed method enables more flexible generative modeling than WGAN. Generative Adversarial Networks (GANs) BID2 are a powerful framework of generative modeling which is formulated as a minimax game between two networks: A generator network generates fake-data from some noise source and a discriminator network discriminates between fake-data and real-data. GANs can generate much more realistic images than other generative models like variational autoencoder BID10 or autoregressive models BID14, and have been widely used in high-resolution image generation BID8, image inpainting BID18, image-to-image translation BID7, to mention a few. However, GANs are often hard to train, and various ways to stabilize training have been proposed by many recent works. Nonetheless, consistently stable training of GANs remains an open problem. GANs employ the Jensen-Shannon (JS) divergence to measure the distance between the distributions of real-data and fake-data BID2. provided an analysis of various distances and divergence measures between two probability distributions in view of their use as loss functions of GANs, and proposed Wasserstein GAN (WGAN) which has better theoretical properties than the original GANs. WGAN requires that the discriminator (called the critic in) must lie within the space of 1-Lipschitz functions to evaluate the Wasserstein distance via the Kantorovich-Rubinstein dual formulation. further proposed implementing the critic with a deep neural network and applied weight clipping in order to ensure that the critic satisfies the Lipschitz condition. However, weight clipping limits the critic's function space and can cause gradients in the critic to explode or vanish if the clipping parameters are not carefully chosen BID3. WGAN-GP BID3 and Spectral Normalization (SN) BID12 apply regularization and normalization, respectively, on the critic trying to make the critic 1-Lipschitz, but they fail to optimize the true Wasserstein distance. In the latest work, BID11 proposed a new WGAN variant to evaluate the exact empirical Wasserstein distance. They evaluate the empirical Wasserstein distance between the empirical distributions of real-data and fake-data in the discrete case of the Kantorovich-Rubinstein dual for-mulation, which can be solved efficiently because the dual problem becomes a finite-dimensional linear-programming problem. The generator network is trained using the critic network learnt to approximate the solution of the dual problem. However, the problem of approximation error by the critic network remains. In this paper, we propose a new generative model without the critic, which learns by directly evaluating gradient of the exact empirical optimal transport cost in the primal domain. The proposed method corresponds to stochastic gradient descent of the optimal transport cost. argued that JS divergences are potentially not continuous with respect to the generator's parameters, leading to GANs training difficulty. They proposed instead using the Wasserstein-1 distance W 1 (q, p), which is defined as the minimum cost of transporting mass in order to transform the distribution q into the distribution p. Under mild assumptions, W 1 (q, p) is continuous everywhere and differentiable almost everywhere. The WGAN objective function is constructed using the Kantorovich-Rubinstein duality (, Chapter 5) as DISPLAYFORM0 to obtain min DISPLAYFORM1 where D is the set of all 1-Lipschitz functions, where P r is the real-data distribution, and where P g is the generator distribution implicitly defined by y = G(z), z ∼ p(z). Minimization of this objective function with respect to G with optimal D is equivalent to minimizing W 1 (P r, P g). et al. further proposed implementing the critic D in terms of a deep neural network with weight clipping. Weight clipping keeps the weight parameter of the network lying in a compact space, thereby ensuring the desired Lipschitz condition. For a fixed network architecture, however, weight clipping may significantly limit the function space to a quite small fraction of all possible 1-Lipschitz functions representable by networks with the prescribed architecture. BID3 proposed introduction of gradient penalty (GP) to the WGAN objective function in place of the 1-Lipschitz condition in the Kantorovich-Rubinstein dual formulation, in order to explicitly encourage the critic to have gradients with magnitude equal to 1. Since enforcing the constraint of unit-norm gradient everywhere is intractable, they proposed enforcing the constraint only along straight line segments, each connecting a real-data point and a fake-data point. The ing learning scheme, which is called the WGAN-GP, was shown to perform well experimentally. It was pointed out, however BID12, that WGAN-GP is susceptible to destabilization due to gradual changes of the support of the generator distribution as learning progresses. Furthermore, the critic can easily violate the Lipschitz condition in practice, so that there is no guarantee that WGAN-GP optimizes the true Wasserstein distance. SN, proposed by BID12, is based on the observation that the Lipschitz norm of a critic represented by a multilayer neural network is bounded from above by the product, across all layers, of the Lipschitz norms of the activation functions and the spectral norms of the weight matrices, and normalizes each of the weight matrices with its spectral norm to ensure the ing critic to satisfy the desired Lipschitz condition. It is well known that, for any m × n matrix W = (w ij), the max norm W max = max{|w ij |} and the spectral norm σ(W) satisfy the inequality DISPLAYFORM0 The proposed method in this paper is based on the fact that the optimal transport cost between two probability distributions can be evaluated efficiently when the distributions are uniform over finite sets of the same cardinality. Our proposal is to evaluate empirical optimal transport costs on the basis of equal-size sample datasets of real-and fake-data points. The optimal transport cost between the real-data distribution P r and the generator distribution P g is defined as DISPLAYFORM0 where c(x, y) is the cost of transporting one unit mass from x to y, assumed differentiable with respect to its arguments almost everywhere, and where Π(P r, P g) denotes the set of all couplings between P r and P g, that is, all joint probability distributions that have marginals P r and P g.Let D = {x j |x j ∼ P r (x)} be a dataset consisting of independent and identically-distributed (iid) real-data points, and F = {y i |y i ∼ P g (y)} be a dataset consisting of iid fake-data points sampled from the generator. Let P D and P F be the empirical distributions defined by the datasets D and F, respectively. We further assume in the following that |D| = |F | = N holds. The empirical optimal transport costĈ(D, F) = C(P D, P F) between the two datasets D and F is formulated as a linear-programming problem, aŝ DISPLAYFORM1 DISPLAYFORM2 DISPLAYFORM3 It is known BID15 that the linear-programming problem- FORMULA6 admits solutions which are permutation matrices. One can then replace the constraints M i,j ≥ 0 in FORMULA6 with M i,j ∈ {0, 1} without affecting the optimality. The ing optimization problem is what is called the linear sum assignment problem, which can be solved more efficiently than the original linear-programming problem. As far as the authors' knowledge, the most efficient algorithm to date for solving a linear sum assignment problem has time complexity of O(N 2.5 log(N C)), where C = max i,j c(x j, y i) when one scales up the costs {c(x j, y i)|x j ∈ D, y i ∈ F } to integers (, Chapter 4). This is a problem to find the optimal transport plan, where M i,j = 1 is corresponding to transporting fake-data point y i ∈ F to real-data point x j ∈ D, and where the objective is to minimize the average FIG0 shows a two-dimensional example of this problem and its solution. - FORMULA6 with N = 8. Circles • represent real-data points in D and triangles represent fake-data points in F. Arrows between circles • and filled triangles show the optimal transport plan M * with c(x, y) = x − y 2, which is an identity matrix. Arrows between open △ and filled triangles show small perturbations of F, which do not change M *. DISPLAYFORM4 One requires evaluations not only of the optimal transport cost C(P r, P g) but also of its derivative in order to perform learning of the generator with backpropagation. Let θ denote the parameter of the generator, and let ∂ θ C denote the derivative of the optimal transport cost C with respect to θ. Conditional on z, the generator output G(z) is a function of θ. Hence, in order to estimate ∂ θ C, in our framework one has to evaluate ∂ θĈ. In general, it is difficult to differentiate with respect to generator output y i, as the optimal transport plan M * can be highly dependent on y i. Under the assumption |D| = |F | = N which we adopt here, however, the feasible set for M is the set of all permutation matrices and is a finite set. It then follows that, as a generic property, the optimal transport plan M * is unchanged under small enough perturbations of F (see FIG0 . We take advantage of this fact and regard M * as independent of y i . Now that differentiation of becomes tractable, we use as the loss function of the generator and update the generator with the direct gradient of the empirical optimal transport cost, as DISPLAYFORM5 Although the framework described so far is applicable to any optimal transport cost, several desirable properties can be stated if one specializes in the Wasserstein distance. Assume, for a given p ≥ 1, that the real-data distribution P r and the generator distribution P g have finite moments of order p. The Wasserstein-p distance between P r and P g is defined in terms of the optimal transport cost with c(x, y) = x − y p as DISPLAYFORM6 Due to the law of large numbers, the empirical distributions P D and P F converge weakly to P r and P g, respectively, as N → ∞. It is also known (, Theorem 6.9) that the Wasserstein-p distance W p metrizes the space of probability measures with finite moments of order p. Consequently, the empirical Wasserstein distanceŴ p (D, F) is a consistent estimator of the true Wasserstein distance W p (P r, P g). Furthermore, with the upper bound of the error of the estimator DISPLAYFORM7 which is derived on the basis of the triangle inequality, as well as with the upper bounds available for expectations of W p (P D, P r) and W p (P F, P g) under mild conditions BID17, one can see thatŴ p (D, F) is an asymptotically unbiased estimator of W p (P r, P g).Note that our method can directly evaluate the empirical Wasserstein distance without recourse to the Kantorovich-Rubinstein dual. Hence, our method does not use a critic and is therefore no longer a GAN. It is also applicable to any optimal transport cost. We summarize the proposed method in Algorithm 1. We first show experimental on the MNIST dataset of handwritten digits. In this experiment, we resized the images to resolution 64 × 64 so that we can use the convolutional neural networks Sample {x i} i∈{1,...,N} ∼ X real from real-data. Sample {z j} j∈{1,...,N} ∼ p(z) from random noises. Let y j = G θ (z j), ∀j ∈ {1, . . ., N}. Solve-7 to obtain M *.9: DISPLAYFORM0 θ ← Adam(g θ, θ, α, β 1, β 2) 11: end while described in Appendix A.1 as the critic and the generator. In all methods, the batch size was set to 64 and the prior noise distribution was the 100-dimensional standard normal distribution. The maximum number of iterations in training of the generator was set to 30,000. The Wasserstein-1 distance with c(x, y) = x − y 1 was used. More detailed settings are described in Appendix B.1.Although several performance metrics have been proposed and are commonly used to evaluate variants of WGAN, we have decided to use the empirical Wasserstein distance (EWD) to compare performance of all methods. It is because all the methods adopt objective functions that are based on the Wasserstein distance, and because EWD is a consistent and asymptotically unbiased estimator of the Wasserstein distance and can efficiently be evaluated, as discussed in Section 4. TAB1 shows EWD evaluated with 256 samples and computation time per generator update for each method. For reference, performance comparison with the Fréchet Inception Distance BID4 and the Inception Score BID13, which are commonly used as performance measures to evaluate GANs using feature space embedding with an inception model, is shown in Appendix C. The proposed method achieved a remarkably small EWD and computational cost compared with the variants of WGAN. Our method required the lowest computational cost in this experimental setting mainly because it does not use the critic. Although we think that the batch size used in the experiment of the proposed method was appropriate since the proposed method achieved lower EWD, if a larger batch size would be required in training, it will take much longer time to solve the linear sum assignment problem-.We further investigated behaviors of the methods compared in more detail, on the basis of EWD. WGAN-SN failed to learn. The loss function of the critic showed divergent movement toward −∞, and the behaviors of EWD in different trials were different even though the behaviors of the critic loss were the same (Figure 2 (a) and (b)). WGAN training never failed in 5 trials, and EWD improved stably without sudden deterioration. Although training with WGAN-GP proceeded favorably in initial stages, at certain points the gradient penalty term started to increase, causing EWD to deteriorate (Figure 2 (c) ). This happened in all 5 trials. Since gradient penalty is a weaker restriction than weight clipping, the critic may be more likely to cause extreme behaviors. We examined both WGAN-TS with and without weight scaling. Whereas WGAN-TS with weight scaling did not fail in training but achieved higher EWD than WGAN, WGAN-TS without weight scaling achieved lower EWD than WGAN at the cost of the stability of training (Figure 3). The proposed method was trained stably and never failed in 5 trials. As mentioned in Section 3, the critic in WGAN-TS simply regresses the optimizer of the empirical version of the Kantorovich-Rubinstein dual. Thus, there is no guarantee that the critic will satisfy the 1-Lipschitz condition. BID11 pointed out that it is indeed practically problematic with WGAN-TS, and proposed weight scaling to ensure that the critic satisfies the desired condition. We have empirically found, however, that weight scaling exhibited the following trade-off (Figure 3). Without weight scaling, training of WGAN-TS suddenly deteriorated in some trials because the critic came to not satisfy the Lipschitz condition. With weight scaling, on the other hand, the regression error of the critic with respect to the solution increased and the EWD became worse. The proposed method directly solves the empirical version of the optimal transport problem in the primal domain, so that it is free from such trade-off. Figure 4 shows fake-data images generated by the generators trained with WGAN, WGAN-GP, WGAN-TS, and the proposed method. Although one can identify the digits for the generated images with the proposed method most easily, these images are less sharp. Among the generated images with the other methods, one can notice several images which have almost the same appearance as real-data images, whereas in the proposed method, such fake-data images are not seen and images that seem averaged real-data images belonging to the same class often appear. This might imply that merely minimizing the Wasserstein distance between the real-data distribution and the generator distribution in the raw-image space may not necessarily produce realistic images. We next observed how the generator distribution is updated in order to compare the proposed method with variants of WGAN in terms of the gradients provided. Figure 5 shows typical behavior of the generator distribution trained with the proposed method on the 8-Gaussian toy dataset. The 8-Gaussian toy dataset and experimental settings are described in Appendix B.2. One can observe that, as training progresses, the generator distribution comes closer to the real-data distribution. FIG4 shows comparison of the behaviors of the proposed method, WGAN-GP, and WGAN-TS. We excluded WGAN and WGAN-SN from this comparison: WGAN tended to yield generator distributions that concentrated around a single Gaussian component, and hence training did not progress well. WGAN-SN could not correctly evaluate the Wasserstein distance as in the experiment on the MNIST dataset. One can observe in FIG4 that directions of sample updates are diverse in the proposed method, especially in later stages of training, and that the sample update directions tend to be aligned with the optimal gradient directions. These behaviors will be helpful for the generator to learn the realdata distribution efficiently. In WGAN-GP and WGAN-TS, on the other hand, the sample update directions exhibit less diversity and less alignment with the optimal gradient directions, which would make the generator distribution difficult to spread and would slow training. One would be able to ascribe such behaviors to poor quality of the critic: Those behaviors would arise when the generator learns on the basis of unreliable gradient information provided by the critic without learning sufficiently to accurately evaluate the Wasserstein distance. If one would increase the number n c of critic iterations per generator iteration in order to train the critic better, the total computational cost of training would increase. In fact, n c = 5 is recommended in practice and has commonly been used in WGAN and its variants because the improvement in learning of the critic is thought to be small relative to increase in computational cost. In reality, however, 5 iterations would not be sufficient for the critic to learn, and this might be a principal reason for the critic to provide poor gradient information to the generator in the variants of WGAN. We have proposed a new generative model that learns by directly minimizing exact empirical Wasserstein distance between the real-data distribution and the generator distribution. Since the proposed method does not suffer from the constraints on the transport cost and the 1-Lipschitz constraint imposed on WGAN by solving the optimal transport problem in the primal domain instead of the dual domain, one can construct more flexible generative modeling. The proposed method provides the generator with better gradient information to minimize the Wasserstein distance (Section 5.2) and achieved smaller empirical Wasserstein distance with lower computational cost (Section 5.1) than any other compared variants of WGAN. In the future work, we would like to investigate the behavior of the proposed method when transport cost is defined in the feature space embedded by an appropriate inception model. A NETWORK ARCHITECTURES A.1 CONVOLUTIONAL NEURAL NETWORKS We show in TAB2 the network architecture used in the experiment on the MNIST dataset in Section 5.1. The generator network receives a 100-dimensional noise vector generated from the standard normal distribution as an input. The noise vector is passed through the fully-connected layer and reshaped to 4 × 4 feature maps. Then they are passed through four transposed convolution layers with 5 × 5 kernels, stride 2 and no biases (since performance was empirically almost the same with or without biases, we took the simpler option of not considering biases), where the resolution of feature maps is doubled and the number of them is halved except for the last layer. The critic network is basically the reverse of the generator network. A convolution layer is used instead of a transposed convolution layer in the critic. After the last convolution layer, the feature maps are flattened into a vector and passed through the fully-connected layer. We employed batch normalization BID6 in all intermediate layers in both of the generator and the critic. Rectified linear unit (ReLU) was used as the activation function in all but the last layers. As the activation function in the last layer, the hyperbolic tangent function and the identity function were used for the generator and for the critic, respectively. We show in Table 3 the network architecture used in the experiment on the 8-Gaussian toy dataset in Section 5.2. The generator network architecture receives a 100-dimensional noise vector as in the experiment on the MNIST dataset. The noise vector is passed through the four fully-connected layers with biases and mapped to a two-dimensional space. The critic network is likewise the reverse of the generator network. The MNIST dataset of handwritten digits used in the experiment in Section 5.1 contains 60,000 two-dimensional images of handwritten digits with resolution 28 × 28.We used default parameter settings decided by the proposers of the respective methods. We used RMSProp BID5 with learning rate 5e−5 for the critic and the generator in WGAN. The weight clipping parameter c was set to 0.01. We used Adam BID9 with learning rate 1e−4, β 1 = 0.5, β 2 = 0.999 in the other methods. λ gp in WGAN-GP was set to 10. In the methods with the critic, the number n c of critic iterations per generator iteration was set to 5. The 8-Gaussian toy dataset used in the experiment in Section 5.2 is a two-dimensional synthetic dataset, which contains real-data sampled from the Gaussian mixture distribution with 8 centers equally distant from the origin and unit variance as the real-data distribution. The centers of the 8 Gaussian component distributions are (±10, 0), (0, ±10), and (±10/ √ 2, ±10/ √ 2). 30, 000 samples were generated in advance before training and were used as the real-data samples. In all methods, the batch size was set to 64 and the maximum number of iterations in training the generator was set to 1,000. WGAN and WGAN-SN could not learn well with this dataset, even though we considered several parameter sets. We used Adam with learning rate 1e−3, β 1 = 0.5, β 2 = 0.999 for WGAN-GP, WGAN-TS and the proposed method. λ gp in WGAN-GP was set to 10. In the methods with the critic, the number n c of critic iterations was set to 5. All the numerical experiments in this paper were executed on a computer with an Intel Core i7-6850K CPU (3.60 GHz, 6 cores) and 32 GB RAM, and with four GeForce GTX 1080 graphics cards installed. Linear sum assignment problems were solved using the Hungarian algorithm, which has time complexity of O(N 3). Codes used in the experiments were written in tensorflow 1.10.1 on python 3.6.0, with eager execution enabled. We show the of evaluation of the experimented methods with FID and IS in TAB3. Both FID and IS are commonly used to evaluate GANs. FID calculates the distance between the set of real-data points and that of fake-data points. The smaller the distance is, the better the fake-data points are judged. Assuming that the vector obtained from a fake-or real-data point through the inception model follows a multivariate Gaussian distribution, FID is defined by the following equation: DISPLAYFORM0 where (µ i, Σ i) is the mean vector and the covariance matrix for dataset i, evaluated in the feature space embedded with inception scores. It is nothing but the square of the Wasserstein-2 distance between two multivariate Gaussian distributions with parameters (µ 1, Σ 1) and (µ 2, Σ 2), respectively. IS is a metric to evaluate only the set of fake-data points. Let x i be a data point, y be the label of x i in the data identification task for which the inception model was trained, p(y|x i) be the probability of label y obtained by inputting x i to the inception model. Letting X be the set of all data points used for calculating the score, the marginal probability of label y is p(y) = 1 |X| xi∈X p(y|x i). IS is defined by the following equation: DISPLAYFORM1 where KL is Kullback-Leibler divergence. IS is designed to be high as the data points are easy to identify by the inception model and variation of labels identified from the data points is abundant. In WGAN-GP, WGAN-SN and WGAN-TS*, we observed that training suddenly deteriorated in some trials. We thus used early stopping on the basis of EWD, and the of these methods shown in TAB3 are with early stopping. The proposed method marked the worst in FID and the best in IS among all the methods compared. Certainly, the fake-data generated by the proposed method are non-sharp and do not resemble realdata points, but it seems that it is easy to distinguish them and they have diversity as digit images. If one wishes to produce higher FID using the proposed method, transport cost should be considered in the desired space corresponding to FID.

We have proposed a flexible generative model that learns stably by directly minimizing exact empirical Wasserstein distance.

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

An empirical study of variational inference based on chi-square divergence minimization, showing that minimizing the CUBO is trickier than maximizing the ELBO


Question: Represent the article for finding a single-sentence TLDR

Question: Neural Architecture Search (NAS) is an exciting new field which promises to be as much as a game-changer as Convolutional Neural Networks were in 2012. Despite many great works leading to substantial improvements on a variety of tasks, comparison between different methods is still very much an open issue. While most algorithms are tested on the same datasets, there is no shared experimental protocol followed by all. As such, and due to the under-use of ablation studies, there is a lack of clarity regarding why certain methods are more effective than others. Our first contribution is a benchmark of 8 NAS methods on 5 datasets. To overcome the hurdle of comparing methods with different search spaces, we propose using a method’s relative improvement over the randomly sampled average architecture, which effectively removes advantages arising from expertly engineered search spaces or training protocols. Surprisingly, we find that many NAS techniques struggle to significantly beat the average architecture baseline. We perform further experiments with the commonly used DARTS search space in order to understand the contribution of each component in the NAS pipeline. These experiments highlight that: (i) the use of tricks in the evaluation protocol has a predominant impact on the reported performance of architectures; (ii) the cell-based search space has a very narrow accuracy range, such that the seed has a considerable impact on architecture rankings; (iii) the hand-designed macrostructure (cells) is more important than the searched micro-structure (operations); and (iv) the depth-gap is a real phenomenon, evidenced by the change in rankings between 8 and 20 cell architectures. To conclude, we suggest best practices, that we hope will prove useful for the community and help mitigate current NAS pitfalls, e.g. difficulties in reproducibility and comparison of search methods. The code used is available at https://github.com/antoyang/NAS-Benchmark. As the deep learning revolution helped us move away from hand crafted features and reach new heights , so does Neural Architecture Search (NAS) hold the promise of freeing us from hand-crafted architectures, which requires tedious and expensive tuning for each new task or dataset. Identifying the optimal architecture is indeed a key pillar of any Automated Machine Learning (AutoML) pipeline. Research in the last two years has proceeded at a rapid pace and many search strategies have been proposed, from Reinforcement Learning , to Evolutionary Algorithms , to Gradient-based methods ). Still, it remains unclear which approach and search algorithm is preferable. Typically, methods have been evaluated on accuracy alone, even though accuracy is influenced by many other factors besides the search algorithm. Comparison between published search algorithms for NAS is therefore either very difficult (complex training protocols with no code available) or simply impossible (different search spaces), as previously pointed out (; ;). NAS methods have been typically decomposed into three components : search space, search strategy and model evaluation strategy. This division is important to keep in mind, as an improvement in any of these elements will lead to a better final performance. But is a method with a more (manually) tuned search space a better AutoML algorithm? If the key idea behind NAS is to find the optimal architecture, without human intervention, why are we devoting so much energy to infuse expert knowledge into the pipeline? Furthermore, the lack of ablation studies in most works makes it harder to pinpoint which components are instrumental to the final performance, which can easily lead to Hypothesizing After the Results are Known (HARKing;). Paradoxically, the huge effort invested in finding better search spaces and training protocols, has led to a situation in which any randomly sampled architecture performs almost as well as those obtained by the search strategies. Our findings suggest that most of the gains in accuracy in recent contributions to NAS have come from manual improvements in the training protocol, not in the search algorithms. As a step towards understanding which methods are more effective, we have collected code for 8 reasonably fast (search time of less than 4 days) NAS algorithms, and benchmarked them on 5 well known CV datasets. Using a simple metric-the relative improvement over the average architecture of the search space-we find that most NAS methods perform very similarly and rarely substantially above this baseline. The methods used are DARTS, StacNAS, PDARTS, MANAS, CNAS, NSGANET, ENAS and NAO. The datasets used are CIFAR10, CIFAR100, SPORT8, MIT67 and FLOWERS102. Through a number of additional experiments on the widely used DARTS search space, we will show that: (a) how you train your model has a much bigger impact than the actual architecture chosen; (b) different architectures from the same search space perform very similarly, so much so that (c) hyperparameters, like the number of cells, or the seed itself have a very significant effect on the ranking; and (d) the specific operations themselves have less impact on the final accuracy than the hand-designed macro-structure of the network. Notably, we find that the 200+ architectures sampled from this search space (available from the link in the abstract) are all within a range of one percentage point (top-1 accuracy) after a standard full training on CIFAR10. Finally, we include some observations on how to foster reproducibility and a discussion on how to potentially avoid some of the encountered pitfalls. As mentioned, NAS methods have the potential to truly revolutionize the field, but to do so it is crucial that future research avoids common mistakes. Some of these concerns have been recently raised by the community. For example, highlight that most NAS methods a) fail to compare against an adequate baseline, such as a properly implemented random search strategy, b) are overly complex, with no ablation to properly assign credit to the important components, and c) fail to provide all details needed for successfully reproducing their . In our paper we go one step further and argue that the relative improvement over the average (randomly sampled) architecture is an useful tool to quantify the effectiveness of a proposed solution and compare it with competing methods. To partly answer their second point, and understand how much the final accuracy depends on the specific architecture, we implement an in-depth study of the widely employed DARTS search space and perform an ablation on the commonly used training techniques (e.g. Cutout, DropPath, AutoAugment). In addition, also took the important step of systematically using fair baselines, and suggest random search with early stopping, averaged over multiple seeds, as an extremely competitive baseline. They find that the search spaces of three methods investigated (DARTS, ENAS, NAO) have been expertly engineered to the extent that any randomly selected architecture performs very well. In contrast, we show that even random sampling (without search) provides an incredibly competitive baseline. Our relative improvement metric allows us to isolate the contribution of the search strategy from the effects of the search space and training pipeline. Thus, we further confirm the authors' claim, showing that indeed the average architecture performs extremely well and that how you train a model has more impact than any specific architecture. In this section we present a systematic evaluation of 8 methods on 5 datasets using a strategy that is designed to reveal the quality of each method's search strategy, removing the effect of the manuallyengineered training protocol and search space. The goal is to find general trends and highlight common features rather than just pin-pointing the most accurate algorithm. Understanding why methods are effective is not an easy task: most introduce variations to previous search spaces, search strategies, and training protocols-with ablations disentangling the contribution of each component often incomplete or missing. In other words, how can we be sure that a new state-of-the-art method is not so simply due to a better engineered search space or training protocol? To address this issue we compare a set of 8 methods with randomly sampled architectures from their respective search spaces, and trained with the same protocol as the searched architectures. The ultimate goal behind NAS should be to return the optimal model for any dataset given, at least within the limits of a certain task, and we feel that the current practices of searching almost exclusively on CIFAR10 go against this principle. Indeed, to avoid the very concrete risk of overfitting to this set of data, NAS methods should be tested on a variety of tasks. For this reason we run experiments on 5 different datasets. Criteria for dataset selection. We selected datasets to cover a variety of subtasks within image classification. In addition to the standard CIFAR10 we select CIFAR100 for a more challenging object classification problem ; SPORT8 for action classification ; MIT67 for scene classification ; and FLOWERS102 for finegrained object classification . More details are given in the Appendix. Criteria for method selection. We selected methods which (a) have open-source code, or provided it upon request, and (b) have a reasonable running time, specifically a search time under 4 GPU-days on CIFAR10. The selected methods are: DARTS, StacNAS, PDARTS , MANAS , CNAS , NSGANET , ENAS , and NAO . With the exception of NAO and NSGANET, all methods are DARTS variants and use weight sharing. Evaluation protocol. NAS algorithms usually consist of two phases: (i) search, producing the best architecture according to the search algorithm used; (ii) augmentation, consisting in training from scratch the best model found in the search phase. We evaluate methods as follows: 1. Sample 8 architectures from the search space, uniformly at random, and use the method's code to augment these architectures (same augment seed for all); 2. Use the method's code to search for 8 architectures and augment them (different search seed, same augment seed); 3. Report mean and standard deviation of the top-1 test accuracy, obtained at the end of the augmentation, for both the randomly sampled and the searched architectures; Since both learned and randomly sampled architectures share the same search space and training protocol, calculating a relative improvement over this random baseline as RI = 100 × (Acc m − Acc r)/Acc r can offer insights into the quality of the search strategy alone. Acc m and Acc r represent the top-1 accuracy of the search method and random sampling strategies, respectively. A good, general-purpose NAS method is expected to yield RI > 0 consistently over different searches and across different subtasks. We emphasize that the comparison is not against random search, but rather against random sampling, i.e., the average architecture of the search space. For example, in the DARTS search space, for each edge in the graph that defines a cell we select one out of eight possible operations (e.g. pooling or convolutions) with uniform probability 1/8. Hyperparameters are optimized on CIFAR10, according to the values reported by the corresponding authors. Since most methods do not include their optimization as part of the search routine, we assumed them to be robust and generalizable to other tasks. As such, aside from scaling down the architecture depending on dataset size, experiments on other datasets use the same hyperparameters. Other training details and references are given in the Appendix. Figure 1 shows the evaluation on the 5 datasets, from which we draw two main . First, the improvements over random sampling tend to be small. In some cases the average performance of a method is even below the average randomly sampled architecture, which suggests that the search methods are not converging to desirable architectures. Second, the small range of accuracies obtained hints at narrow search spaces, where even the worst architectures perform reasonably well. See Section 5 for more experiments corroborating this . We observe also that, on CIFAR10, the top half of best-performing methods (PDARTS, MANAS, DARTS, StacNAS) all perform similarly and positively in relation to their respective search spaces, but more variance is seen on the other datasets. This could be explained by the fact that most methods' hyperparameters have been optimized on CIFAR10 and might not generalize as well on different datasets. As a matter of fact, we found that all NAS methods neglect to report the time needed to optimize hyperparameters. In addition, Table 1 shows the relative improvement metric RI (see intro to Section 3) for each method and dataset. The computational cost of searching for architectures is a limiting factor in their applicability and, therefore, an important variable in the evaluation of NAS algorithms. Figure 2 shows the performance as well and the computational cost of the search phase on CIFAR10. In this section we attempt to shed some light on the surprising of the previous section. We noticed that there was a much larger differences between the random baselines of different methods than the actual increase in performance of each approach. We hypothesized that how a network is trained (the training protocol) has a larger impact on the final accuracy than which architecture is trained, for each search space. To test this, we performed sensitivity analysis using the most common performanceboosting training protocols. We decided to evaluate architectures from the commonly used DARTS search space on the CI-FAR10 dataset. We use the following process: 1) sample 8 random architectures, 2) train them with different training protocols (details below) and 3) report mean, standard deviation and maximum of the top-1 test accuracy at the end of the training process. Training protocols. The simplest training protocol, which we will call Base is similar to the one used in DARTS, but with all tricks disabled: the model is simply trained for 600 epochs. On the other extreme, our full protocol uses several tricks which have been used in recent works (b;) et al., 2018), extended training for 1500 epochs (1500E), and increased number of channels (50C). In between these two extremes, by selectively enabling and disabling each component, we evaluated a further 8 intermediate training protocols. When active, DropPath probability is 0.2, cutout length is 16, auxiliary tower weight is 0.4, and AutoAugment combined with Cutout are used after standard data pre-processing techniques previously described, as in. As shown in Figure 3, a large difference of over 3 percentage points (p.p.) exists between the simplest and the most advanced training protocols. Indeed, this is much higher than any improvement over random sampling observed in the previous section: for example, on CIFAR10, the best improvement observed was 0.69 p.p. In other words, the training protocol is often far more important than the architecture used. Note that the best accuracy of the 8 random architectures training with the best protocol is 98.15%, which is only 0.25 p.p. below state-of-the-art . To summarize, it seems that most recent state-of-the-art , though impressive, cannot always be attributed to superior search strategies. Rather, they are often the of expert knowledge applied to the evaluation protocol. In Figure 10 (Appendix A.3.1) we show similar when training a ResNet-50 with the same protocols. Kendall tau rank correlation non-smoothed accuracy smoothed accuracy (w=20) Figure 5: Correlation between accuracies at different epochs and final accuracy, using raw and smoothed accuracies over a window w. To better understand the from the previous section, we sampled a considerable number of architectures (200+) from the most commonly used search space and fully trained them with the matching training protocol (Cutout+DropPath+Auxiliary Towers). This allows us to get a sense of how much variance exists between the different models (training statistics are made available at the link in the abstract). As we can observe from Figure 4, architectures sampled from this search space all perform similarly, with a mean of 97.03 ± 0.23. The worst architecture we found had an accuracy of 96.18, while the best achieved 97.56. To put this into perspective, many methods using the same training protocol, fall within (or very close to) the standard deviation of the average architecture. Furthermore, as we can observe in Figure 7, the number of cells (a human-picked hyperparameter) has a much larger impact on the final accuracy. In Figure 5 we used the training statistics of the 214 models to plot the correlation between test accuracies at different epochs: it grows slowly in an almost linear fashion. We note that using the moving average of the accuracies yields a stronger correlation, which could be useful for methods using early stopping. To test whether the from the previous section were due to the choice of available operations, we developed an intentionally sub-optimal search space containing 4 plain convolutions (1×1, 3×3, 7 × 7, 11 × 11), 2 max pooling operators (3 × 3, 5 × 5) plus the none and skip connect operations. This proposed search space is clearly more parameter inefficient compared to the commonly used DARTS one (which uses both dilated and separable ones), and we expect it to perform worse. We sampled 56 architectures from this new search space and trained them with the DARTS training protocol (Cutout+DropPath+Auxiliary Towers), for fair comparison with the from the previous section. Figure 6 shows the ing histogram, together with the one obtained from the classical DARTS space of operations. The two distributions are only shifted by 0.18 accuracy points. Given the minor difference in performance, the specific operations are not a key ingredient in the success of this search space. Very likely, it's the well engineered cell structure that allows the model to perform as well as it does. A necessary practice for many weight-sharing methods ) is to restart the training from scratch after the search phase, with a different number of cells. Recent works have warned that this procedure might negatively affect ranking; similarly, the role of the seed has been previously recognized as as a fundamental element in reproducibility . To test the impact of seed, we randomly sampled 32 architectures and trained them with two different seeds (Figure 8). Ranking is heavily influenced, as the Kendall tau correlation between the two sets of training is 0.48. On average, the test accuracy changes by 0.13% ± 0.08 (max change is 0.39%), which is substantial considering the small gap between random architectures and NAS methods. To test the depth-gap we trained another 32 with different number of cells (Figure 9). The correlation between the two different depths is not very strong as measured by Kendall Tau (0.54), with architectures shifting up and down the rankings by up to 18 positions (out of 32). Methods employing weight sharing (WS) would see an even more pronounced effect as the architectures normally chosen at 8 cells would have been training sub-optimally due to the WS itself . These findings point towards two issues. The first is that since the seed has such a large effect on ranking, it stands to reason that the final accuracy reported should be averaged over multiple seeds. The second is that, if the lottery ticket hypothesis holds-so that specific sub-networks are better mainly due to their lucky initialization; Frankle & Carbin., 2018-together with our findings, this could be an additional reason why methods searching on a different number of cells than the final model, struggle to significantly improve on the average randomly sampled architecture. In this section we offer some suggestions on how to mitigate the issues in NAS research. Augmention tricks: while achieving higher accuracies is clearly a desirable goal, we have shown in section 4, that using well engineered training protocols can hide the contribution of the search algorithm. We therefore suggest that both , with and without training tricks, should be reported. An example of best practice is found in. Search Space: it is difficult to evaluate the effectiveness of any given proposed method without a measure of how good randomly sampled architectures are. This is not the same thing as performing a random search which is a search strategy in itself; random sampling is simply used to establish how good the average model is. A simple approach to measure the variability of any new given search space could be to randomly sample k architectures and report mean and standard deviation. We hope that future works will attempt to develop more expressive search spaces, capable of producing both good and bad network designs. Restricted search spaces, while guaranteeing good performance and quick , will inevitably be constrained by the bounds of expert knowledge (local optima) and will be incapable of reaching more truly innovative solutions (closer to the global optima). As our findings in section 5.2 suggest, the overall wiring (the macro-structure) is an extremely influential component in the final performance. As such, future research could investigate the optimal wiring at a global level: an interesting work in this direction is Xie et al. (2019a). Multiple datasets: as the true goal of AutoML is to minimize the need for human experts, focusing the research efforts on a single dataset will inevitably lead to algorithmic overfitting and/or methods heavily dependent on hyperparameter tuning. The best solution for this is likely to test NAS algorithms on a battery of datasets, with different characteristics: image sizes, number of samples, class granularity and learning task. Investigating hidden components: as our experiments in Sections 4 and 5.2 show, the DARTS search space is not only effective due to specific operations that are being chosen, but in greater part due to the overall macro-structure and the training protocol used. We suggest that proper ablation studies can lead to better understanding of the contributions of each element of the pipeline. The importance of reproducibility: reproducibility is of extreme relevance in all sciences. To this end, it is very important that authors release not only their best found architecture but also the corresponding seed (if they did not average over multiple ones), as well as the code and the detailed training protocol (including hyperparameters). To this end, NAS-Bench-101 , a dataset mapping architectures to their accuracy, can be extremely useful, as it allows the quality of search strategies to be assessed in isolation from other NAS components (e.g. search space, training protocol) in a quick and reproducible fashion. The code for this paper is open-source (link in the abstract). We also open-source the 200+ trained architectures used in Section 5. Hyperparameter tuning cost: tuning hyperparameters in NAS is an extremely costly component. Therefore, we argue that either (i) hyperparameters are general enough so that they do not require tuning for further tasks, or the cost is included in the search budget. AutoML, and NAS in particular, have the potential to truly democratize the use of machine learning for all, and could bring forth very notable improvements on a variety of tasks. To truly step forward, a principled approach, with a focus on fairness and reproducibility is needed. In this paper we have shown that, for many NAS methods, the search space has been engineered such that all architectures perform similarly well and that their relative ranking can easily shift. We have furthermore showed that the training protocol itself has a higher impact on the final accuracy than the actual network. Finally, we have provided some suggestions on how to make future research more robust to these issues. We hope that our findings will help the community focus their efforts towards a more general approach to automated neural architecture design. Only then can we expect to learn from NASgenerated architectures as opposed to the current paradigm where search spaces are heavily influenced by our current (human) expert knowledge. A APPENDIX This section details the datasets and the hyperparameters used for each method on each dataset. Search spaces were naturally left unchanged. Hyperparameters were chosen as close as possible to the original paper and occasionally updated to more recent implementations. The network size was tuned similarly for all methods for SPORT8, MIT67 and FLOWERS102. All experiments were run on NVIDIA Tesla V100 GPUs. During search, models are trained/validated on the training/validation subsets, respectively During the final evaluation, the model is trained on the training+validation subsets and tested on the test subset. Common hyperparameters. All 8 methods share a common number of hyperparameters precised here. When SGD optimizer is used, momentum is.9 while when Adam is used, momentum is β = (0.5, 0.999). Gradient clipping is set at 5. DARTS, StacNAS, PDARTS, MANAS, CNAS common hyperparameters. These methods are inspired by DARTS code-wise and consequently share a common number of hyperparameters, which we precise in table 1. We used an unofficial implementation provided by the authors. The search process consists of 2 stages, of which the details are given in table 1. Additional enhancements are the same as DARTS. PDARTS. We used the official implementation: https://github.com/chenxin061/pdarts. The search process consists of 3 stages, of which general details are given in table 1. At stage 1, 2 and 3 respectively, the number of operations decreases from 8 to 5 to 3, and the dropout probability on skip-connect increases from 0.0 to 0.4 to 0.7 for CIFAR10, SPORT8, MIT67 and FLOWERS102 We present here the datasets used and how they are pre-processed. CIFAR10. The CIFAR10 dataset is a dataset of 10 classes and consists of 50, 000 training images and 10, 000 test images of size 32×32. CIFAR100. The CIFAR100 dataset is a dataset of 100 classes and consists of 50, 000 training images and 10, 000 test images of size 32×32. Each of these datasets is split into a training, validation and testing subsets of size 25, 000, 25, 000 and 10, 000 respectively. For both these datasets, we use standard data pre-processing and augmentation techniques, i.e. subtracting the channel mean and dividing by the channel standard deviation; centrally padding the training images to 40×40 and randomly cropping them back to 32×32; and randomly clipping them horizontally. SPORT8. This is an action recognition dataset containing 8 sport event categories and a total of 1579 images . The tiny size of this dataset stresses the generalization capabilities of any NAS method applied to it. MIT67. This is a dataset of 67 classes representing different indoor scenes and consists of 15, 620 images of different sizes . FLOWERS102. This is a dataset of 102 classes representing different species of flowers and consists of 8, 189 images of different sizes . Each of these datasets is split into a training, validation and testing subsets with proportions 40/40/20 (%). For each one, we use use standard data pre-processing and augmentation techniques, i.e. subtracting the channel mean and dividing the channel standard deviation, cropping the training images to random size and aspect ratio, resizing them to 224×224, and randomly changing their brightness, contrast, and saturation, while resizing test images to 256×256 and cropping them at the center. Figure 3 ). Result are for 8 runs of each training protocol. For ResNet-50, the auxiliary tower was added after layer 2. As DropPath would not have been straightforward to apply, we instead implemented Stocastic Depth , to a similar effect.

Represent the short summary for finding the corresponding scientific article (long)

A study of how different components in the NAS pipeline contribute to the final accuracy. Also, a benchmark of 8 methods on 5 datasets.

Contrary to previous beliefs, the training performance of deep networks, when measured appropriately, is predictive of test performance, consistent with classical machine learning theory.


Question: Represent the article for finding a single-sentence TLDR

Question: Identifying analogies across domains without supervision is a key task for artificial intelligence. Recent advances in cross domain image mapping have concentrated on translating images across domains. Although the progress made is impressive, the visual fidelity many times does not suffice for identifying the matching sample from the other domain. In this paper, we tackle this very task of finding exact analogies between datasets i.e. for every image from domain A find an analogous image in domain B. We present a matching-by-synthesis approach: AN-GAN, and show that it outperforms current techniques. We further show that the cross-domain mapping task can be broken into two parts: domain alignment and learning the mapping function. The tasks can be iteratively solved, and as the alignment is improved, the unsupervised translation function reaches quality comparable to full supervision. Humans are remarkable in their ability to enter an unseen domain and make analogies to the previously seen domain without prior supervision ("This dinosaur looks just like my dog Fluffy"). This ability is important for using previous knowledge in order to obtain strong priors on the new situation, which makes identifying analogies between multiple domains an important problem for Artificial Intelligence. Much of the recent success of AI has been in supervised problems, i.e., when explicit correspondences between the input and output were specified on a training set. Analogy identification is different in that no explicit example analogies are given in advance, as the new domain is unseen. Recently several approaches were proposed for unsupervised mapping between domains. The approaches take as input sets of images from two different domains A and B without explicit correspondences between the images in each set, e.g. Domain A: a set of aerial photos and Domain B: a set of Google-Maps images. The methods learn a mapping function T AB that takes an image in one domain and maps it to its likely appearance in the other domain, e.g. map an aerial photo to a Google-Maps image. This is achieved by utilizing two constraints: (i) Distributional constraint: the distributions of mapped A domain images (T AB (x)) and images of the target domain B must be indistinguishable, and (ii) Cycle constraint: an image mapped to the other domain and back must be unchanged, i.e., T BA (T AB (x)) = x. In this paper the task of analogy identification refers to finding pairs of examples in the two domains that are related by a fixed non-linear transformation. Although the two above constraints have been found effective for training a mapping function that is able to translate between the domains, the translated images are often not of high enough visual fidelity to be able to perform exact matching. We hypothesize that it is caused due to not having exemplar-based constraints but rather constraints on the distributions and the inversion property. In this work we tackle the problem of analogy identification. We find that although current methods are not designed for this task, it is possible to add exemplar-based constraints in order to recover high performance in visual analogy identification. We show that our method is effective also when only some of the sample images in A and B have exact analogies whereas the rest do not have exact analogies in the sample sets. We also show that it is able to find correspondences between sets when no exact correspondences are available at all. In the latter case, since the method retrieves rather than maps examples, it naturally yields far better visual quality than the mapping function. Using the domain alignment described above, it is now possible to perform a two step approach for training a domain mapping function, which is more accurate than the provided by previous unsupervised mapping approaches:1. Find the analogies between the A and B domain, using our method.2. Once the domains are aligned, fit a translation function T AB between the domains y mi = T AB (x i) using a fully supervised method. For the supervised network, larger architectures and non-adversarial loss functions can be used. This paper aims to identify analogies between datasets without supervision. Analogy identification as formulated in this paper is highly related to image matching methods. As we perform matching by synthesis across domains, our method is related to unsupervised style-transfer and image-to-image mapping. In this section we give a brief overview of the most closely related works. Image Matching Image matching is a long-standing computer vision task. Many approaches have been proposed for image matching, most notably pixel-and feature-point based matching (e.g. SIFT BID11). Recently supervised deep neural networks have been used for matching between datasets BID18, and generic visual features for matching when no supervision is available (e.g. BID5). As our scenario is unsupervised, generic visual feature matching is of particular relevance. We show in our experiments however that as the domains are very different, standard visual features (multi-layer VGG-16 BID15) are not able to achieve good analogies between the domains. Generative Adversarial Networks GAN technology presents a major breakthrough in image synthesis (and other domains). The success of previous attempts to generate random images in a class of a given set of images, was limited to very specific domains such as texture synthesis. Therefore, it is not surprising that most of the image to image translation work reported below employ GANs in order to produce realistically looking images. GAN methods train a generator network G that synthesizes samples from a target distribution, given noise vectors, by jointly training a second network D. The specific generative architecture we and others employ is based on the architecture of BID12. In image mapping, the created image is based on an input image and not on random noise BID9 BID21 BID10 BID16.Unsupervised Mapping Unsupervised mapping does not employ supervision apart from sets of sample images from the two domains. This was done very recently BID16 BID9 BID21 for image to image translation and slightly earlier for translating between natural languages BID19. The above mapping methods however are focused on generating a mapped version of the sample in the other domain rather than retrieving the best matching sample in the new domain. Supervised Mapping When provided with matching pairs of (input image, output image) the mapping can be trained directly. An example of such method that also uses GANs is, where the discriminator D receives a pair of images where one image is the source image and the other is either the matching target image ("real" pair) or a generated image ("fake" pair); The link between the source and the target image is further strengthened by employing the U-net architecture of BID14. We do not use supervision in this work, however by the successful completion of our algorithm, correspondences are generated between the domains, and supervised mapping methods can be used on the inferred matches. Recently, BID2 demonstrated improved mapping , in the supervised settings, when employing the perceptual loss and without the use of GANs. In this section we detail our method for analogy identification. We are given two sets of images in domains A and B respectively. The set of images in domain A are denoted x i where i ∈ I and the set image in domain B are denoted y j where j ∈ J. Let m i denote the index of the B domain image y mi that is analogous to x i. Our goal is to find the matching indexes m i for i ∈ I in order to be able to match every A domain image x i with a B domain image y mi, if such a match exists. We present an iterative approach for finding matches between two domains. Our approach maps images from the source domain to the target domain, and searches for matches in the target domain. A GAN-based distribution approach has recently emerged for mapping images across domains. Let x be an image in domain A and y be an image in domain B. A mapping function T AB is trained to map x to T AB (x) so that it appears as if it came from domain B. More generally, the distribution of T AB (x) is optimized to appear identical to that of y. The distributional alignment is enforced by training a discriminator D to discriminate between samples from p(T AB (x)) and samples from p(y), where we use p(x) to denote the distribution of x and p(T AB (x)) to denote the distribution of T AB (x) when x ∼ p(x). At the same time T AB is optimized so that the discriminator will have a difficult task of discriminating between the distributions. The loss function for training T and D are therefore: DISPLAYFORM0 is a binary cross-entropy loss. The networks L D and L T are trained iteratively (as they act in opposite directions).In many datasets, the distribution-constraint alone was found to be insufficient. Additional constraints have been effectively added such as circularity (cycle) BID9 and distance invariance BID0. The popular cycle approach trains one-sided GANs in both the A → B and B → A directions, and then ensures that an A image domain translated to B (T AB (x)) and back to A (T BA (T BA (x))) recovers the original x. Let L 1 denote the L 1 loss. The complete two-sided cycle loss function is given by: DISPLAYFORM1 DISPLAYFORM2 The above two-sided approach yields mapping function from A to B and back. This method provides matching between every sample and a synthetic image in the target domain (which generally does not correspond to an actual target domain sample), it therefore does not provide exact correspondences between the A and B domain images. In the previous section, we described a distributional approach for mapping A domain image x to an image T AB (x) that appears to come from the B domain. In this section we provide a method for providing exact matches between domains. Let us assume that for every A domain image x i there exists an analogous B domain image y mi. Our task is find the set of indices m i. Once the exact matching is recovered, we can also train a fully supervised mapping function T AB, and thus obtain a mapping function of the quality provided by supervised method. Let α i,j be the proposed match matrix between B domain image y j and A domain image x i, i.e., every x i matches a mixture of all samples in B, using weights α i,:, and similarly for y j for a weighing using α:,j of the training samples from A. Ideally, we should like a binary matrix with α i,j = 1 for the proposed match and 0 for the rest. This task is formally written as: DISPLAYFORM0 where L p is a "perceptual loss", which is based on some norm, a predefined image representation, a Laplacian pyramid, or otherwise. See Sec. 3.4.The optimization is continuous over T AB and binary programming over α i,j. Since this is computationally hard, we replace the binary constraint on α by the following relaxed version: DISPLAYFORM1 In order to enforce sparsity, we add an entropy constraint encouraging sparse solutions. DISPLAYFORM2 The final optimization objective becomes: DISPLAYFORM3 The positivity α ≥ 0 and i α i,j = 1 constraints are enforced by using an auxiliary variable β and passing it through a Sof tmax function. DISPLAYFORM4 The relaxed formulation can be optimized using SGD. By increasing the significance of the entropy term (increasing k entropy), the solutions can converge to the original correspondence problem and exact correspondences are recovered at the limit. Since α is multiplied with all mapped examples T AB (x), it might appear that mapping must be performed on all x samples at every batch update. We have however found that iteratively updating T AB for N epochs, and then updating β for N epochs (N = 10) achieves excellent . Denote the β (and α) updates-α iterations and the updates of T AB -T iterations. The above training scheme requires the full mapping to be performed only once at the beginning of the α iteration (so once in 2N epochs). Although the examplar-based method in Sec. 3.2 is in principle able to achieve good matching, the optimization problem is quite hard. We have found that a good initialization of T AB is essential for obtaining good performance. We therefore present AN-GAN -a cross domain matching method that uses both exemplar and distribution based constraints. The AN-GAN loss function consists of three separate constraint types:1. Distributional loss L T dist: The distributions of T AB (x) matches y and T BA (y) matches x (Eq. 3). 2. Cycle loss L T cycle: An image when mapped to the other domain and back should be unchanged (Eq. 4). 3. Exemplar loss L T exemplar: Each image should have a corresponding image in the other domain to which it is mapped (Eq. 11).The AN-GAN optimization problem is given by: min DISPLAYFORM0 The optimization also adversarially trains the discriminators D A and D B as in equation Eq. 6. Initially β are all set to 0 giving all matches equal likelihood. We use an initial burn-in period of 200 epochs, during which δ = 0 to ensure that T AB and T BA align the distribution before aligning individual images. We then optimize the examplar-loss for one α-iteration of 22 epochs, one T -iteration of 10 epochs and another α-iteration of 10 epochs (joint training of all losses did not yield improvements). The initial learning rate for the exemplar loss is 1e − 3 and it is decayed after 20 epochs by a factor of 2. We use the same architecture and hyper-parameters as CycleGAN unless noted otherwise. In all experiments the β parameters are shared between the two mapping directions, to let the two directions inform each other as to likelihood of matches. All hyper-parameters were fixed across all experiments. In the previous sections we assumed a "good" loss function for determining similarity between actual and synthesized examples. In our experiments we found that Euclidean or L 1 loss functions were typically not perceptual enough to provide good supervision. Using the Laplacian pyramid loss as in GLO BID1 does provide some improvement. The best performance was however achieved by using a perceptual loss function. This was also found in several prior works BID4, BID8, BID2.For a pair of images I 1 and I 2, our loss function first extracts VGG features for each image, with the number of feature maps used depending on the image resolution. We use the features extracted by the the second convolutional layer in each block, 4 layers in total for 64X64 resolution images and five layers for 256X256 resolution images. We additionally also use the L 1 loss on the pixels to ensure that the colors are taken into account. Let us define the feature maps for images I 1 and I 2 as φ m 1 and φ m 2 (m is an index running over the feature maps). Our perceptual loss function is: DISPLAYFORM0 Where N P is the number of pixels and N m is the number of features in layer m. We argue that using this loss, our method is still considered to be unsupervised matching, since the features are available off-the-shelf and are not tailored to our specific domains. Similar features have been extracted using completely unsupervised methods (see e.g. BID3 To evaluate our approach we conducted matching experiments on multiple public datasets. We have evaluated several scenarios: (i) Exact matches: Datasets on which all A and B domain images have We compare our method against a set of other methods exploring the state of existing solutions to cross-domain matching:U nmapped − P ixel: Finding the nearest neighbor of the source image in the target domain using L 1 loss on the raw pixels. U nmapped − V GG: Finding the nearest neighbor of the source image in the target domain using VGG feature loss (as described in Sec. 3.4. Note that this method is computationally quite heavy due to the size of each feature. We therefore randomly subsampled every feature map to 32000 values, we believe this is a good estimate of the performance of the method. CycleGAN − P ixel: Train Eqs. 5, 6 using the authors' CycleGAN code. Then use L 1 to compute the nearest neighbor in the target set. CycleGAN − V GG: Train Eqs. 5, 6 using the authors' CycleGAN code. Then use VGG loss to compute the nearest neighbor in the target set. The VGG features were subsampled as before due to the heavy computational cost.α iterations only: Train AN − GAN as described in Sec. 3.3 but with α iterations only, without iterating over T XY .AN − GAN : Train AN − GAN as described in Sec. 3.3 with both α and T XY iterations. We evaluate our method on 4 public exact match datasets:Facades: 400 images of building facades aligned with segmentation maps of the buildings (Radim Tyleček, 2013).Maps: The Maps dataset was scraped from Google Maps by. It consists of aligned Maps and corresponding satellite images. We use the 1096 images in the training set. The original dataset contains around 50K images of shoes from the Zappos50K dataset BID23, (Yu & Grauman). The edge images were automatically detected by using HED (BID20). The original dataset contains around 137k images of Amazon handbags (BID24). The edge images were automatically detected using HED by.For both E2S and E2H the datasets were randomly down-sampled to 2k images each to accommodate the memory complexity of our method. This shows that our method works also for moderately sized dataset. In this set of experiments, we compared our method with the five methods described above on the task of exact correspondence identification. For each evaluation, both A and B images are shuffled prior to training. The objective is recovering the full match function m i so that x i is matched to y mi. The performance metric is the percentage of images for which we have found the exact match in the other domain. This is calculated separately for A → B and B → A.The are presented in Table. 1. Several observations are apparent from the : matching between the domains using pixels or deep features cannot solve this task. The domains used in our experiments are different enough such that generic features are not easily able to match between them. Simple mapping using CycleGAN and matching using pixel-losses does improve matching performance in most cases. CycleGAN performance with simple matching however leaves much space for improvement. The next baseline method matched perceptual features between the mapped source images and the target images. Perceptual features have generally been found to improve performance for image retrieval tasks. In this case we use VGG features as perceptual features as described in Sec. 3.4. We found exhaustive search too computationally expensive (either in memory or runtime) for our datasets, and this required subsampling the features. Perceptual features performed better than pixel matching. We also run the α iterations step on mapped source domain images and target images. This method matched linear combinations of mapped images rather than a single image (the largest α component was selected as the match). This method is less sensitive to outliers and uses the same β parameters for both sides of the match (A → B and B → A) to improve identification. The performance of this method presented significant improvements. The exemplar loss alone should in principle recover a plausible solution for the matches between the domains and the mapping function. However, the optimization problem is in practice hard and did not converge. We therefore use a distributional auxiliary loss to aid optimization. When optimized with the auxiliary losses, the exemplar loss was able to converge through α − T iterations. This shows that the distribution and cycle auxiliary losses are essential for successful analogy finding. Our full-method AN-GAN uses the full exemplar-based loss and can therefore optimize the mapping function so that each source sample matches the nearest target sample. It therefore obtained significantly better performance for all datasets and for both matching directions. In this set of experiments we used the same datasets as above but with M % of the matches being unavailable This was done by randomly removing images from the A and B domain datasets. In this scenario M % of the domain A samples do not have a match in the sample set in the B domain and similarly M % of the B images do not have a match in the A domain.(1 − M)% of A and B images contain exact matches in the opposite domain. The task is identification of the correct matches for all the samples that possess matches in the other domain. The evaluation metric is the percentage of images for which we found exact matches out of the total numbers of images that have an exact match. Apart from the removal of the samples ing in M % of non-matching pairs, the protocol is identical to Sec. 4.1.1.The for partial exact matching are shown in Table. 2. It can be clearly observed that our method is able to deal with scenarios in which not all examples have matches. When 10% of samples do not have matches, are comparable to the clean case. The are not significantly lower for most datasets containing 25% of samples without exact matches. Although in the general case a low exact match ratio lowers the quality of mapping function and decreases the quality of matching, we have observed that for several datasets (notably Facades), AN-GAN has achieved around 90% match rate with as much as 75% of samples not having matches. Although the main objective of this paper is identifying exact analogies, it is interesting to test our approach on scenarios in which no exact analogies are available. In this experiment, we qualitatively evaluate our method on finding similar matches in the case where an exact match is not available. We evaluate on the Shoes2Handbags scenario from BID9. As the CycleGAN architecture is not effective at non-localized mapping we used the DiscoGAN architecture BID9 for the mapping function (and all of the relevant hyper-parameters from that paper).In FIG2 we can observe several analogies made for the Shoes2Handbags dataset. The top example shows that when DiscoGAN is able to map correctly, matching works well for all methods. However in the bottom two rows, we can see examples that the quality of the DiscoGAN mapping is lower. In this case both the DiscoGAN map and DiscoGAN + α iterations present poor matches. On the other hand AN − GAN forced the match to be more relevant and therefore the analogies found by AN − GAN are better. We have shown that our method is able to align datasets with high accuracy. We therefore suggested a two-step approach for training a mapping function between two datasets that contain exact matches but are unaligned: (i) Find analogies using AN − GAN, and (ii) Train a standard mapping function using the self-supervision from stage (i).For the Facades dataset, we were able to obtain 97% alignment accuracy. We used the alignment to train a fully self-supervised mapping function using Pix2Pix. We evaluate on the facade photos to segmentations task as it allows for quantitative evaluation. In Fig. 3 we show two facade photos from the test set mapped by: CycleGAN, Pix2Pix trained on AN-GAN matches DISPLAYFORM0 Figure 3: Supervised vs unsupervised image mapping: The supervised mapping is far more accurate than unsupervised mapping, which is often unable to match the correct colors (segmentation labels). Our method is able to find correspondences between the domains and therefore makes the unsupervised problem, effectively supervised. and a fully-supervised Pix2Pix approach. We can see that the images mapped by our method are of higher quality than CycleGAN and are about the fully-supervised quality. In Table. 3 we present a quantitative comparison on the task. As can be seen, our self-supervised method performs similarly to the fully supervised method, and much better than CycleGAN.We also show for the edges2shoes and edges2handbags datasets. The supervised stage uses a Pix2Pix architecture, but only L 1 loss (rather than the combination with cGAN as in the paper -L 1 only works better for this task). The test set L 1 error is shown in Tab. 4. It is evident that the use of an appropriate loss and larger architecture enabled by the ANGAN-supervision yields improved performance over CycleGAN and is competitive with full-supervision. We have also evaluated our method on point cloud matching in order to test our method in low dimensional settings as well as when there are close but not exact correspondences between the samples in the two domains. Point cloud matching consists of finding the rigid 3D transformation between a set of points sampled from the reference and target 3D objects. The target 3D object is a We ran the experiments using the Bunny benchmark, using the same setting as in BID17. In this benchmark, the object is rotated by a random 3D rotation, and we tested the success rate of our model in achieving alignment for various ranges of rotation angles. For both CycleGAN and our method, the following architecture was used. D is a fully connected network with 2 hidden layers, each of 2048 hidden units, followed by BatchNorm and with Leaky ReLU activations. The mapping function is a linear affine matrix of size 3X3 with a bias term. Since in this problem, the transformation is restricted to be a rotation matrix, in both methods we added a loss term that encourages orthonormality of the weights of the mapper. Namely, W W T − I, where W are the weights of our mapping function. Tab. 5 depicts the success rate for the two methods, for each rotation angle bin, where success is defined in this benchmark as achieving an RMSE alignment accuracy of 0.05.Our significantly outperform the baseline reported in BID17 at large angles. Their are given in graph form, therefore the exact numbers could not be presented in Tab. 5. Inspection of the middle column of Fig.3 in BID17 will verify that our method performs the best for large transformations. We therefore conclude that our method is effective also for low dimensional transformations and well as settings in which exact matches do not exist. We presented an algorithm for performing cross domain matching in an unsupervised way. Previous work focused on mapping between images across domains, often ing in mapped images that were too inaccurate to find their exact matches. In this work we introduced the exemplar constraint, specifically designed to improve match performance. Our method was evaluated on several public datasets for full and partial exact matching and has significantly outperformed baseline methods. It has been shown to work well even in cases where exact matches are not available. This paper presents an alternative view of domain translation. Instead of performing the full operation end-toend it is possible to (i) align the domains, and (ii) train a fully supervised mapping function between the aligned domains. Future work is needed to explore matching between different modalities such as images, speech and text. As current distribution matching algorithms are insufficient for this challenging scenario, new ones would need to be developed in order to achieve this goal.

Finding correspondences between domains by performing matching/mapping iterations

We leverage deterministic autoencoders as generative models by proposing mixing functions which combine hidden states from pairs of images. These mixes are made to look realistic through an adversarial framework.