Question: Represent the article for finding a single-sentence TLDR

Question: Generative Adversarial Networks (GANs) have been shown to produce realistically looking synthetic images with remarkable success, yet their performance seems less impressive when the training set is highly diverse. In order to provide a better fit to the target data distribution when the dataset includes many different classes, we propose a variant of the basic GAN model, a Multi-Modal Gaussian-Mixture GAN (GM-GAN), where the probability distribution over the latent space is a mixture of Gaussians. We also propose a supervised variant which is capable of conditional sample synthesis. In order to evaluate the model's performance, we propose a new scoring method which separately takes into account two (typically conflicting) measures - diversity vs. quality of the generated data. Through a series of experiments, using both synthetic and real-world datasets, we quantitatively show that GM-GANs outperform baselines, both when evaluated using the commonly used Inception Score, and when evaluated using our own alternative scoring method. In addition, we qualitatively demonstrate how the unsupervised variant of GM-GAN tends to map latent vectors sampled from different Gaussians in the latent space to samples of different classes in the data space. We show how this phenomenon can be exploited for the task of unsupervised clustering, and provide quantitative evaluation showing the superiority of our method for the unsupervised clustering of image datasets. Finally, we demonstrate a feature which further sets our model apart from other GAN models: the option to control the quality-diversity trade-off by altering, post-training, the probability distribution of the latent space. This allows one to sample higher quality and lower diversity samples, or vice versa, according to one's needs. Generative models have long been an important and active field of research in machine-learning. Generative Adversarial Networks BID6 include a family of methods for learning generative models where the computational approach is based on game theory. The goal of a GAN is to learn a Generator (G) capable of generating samples from the data distribution (p X), by converting latent vectors from a lower-dimension latent space (Z) to samples in a higher-dimension data space (X). Usually, latent vectors are sampled from Z using the uniform or the normal distribution. In order to train G, a Discriminator (D) is trained to distinguish real training samples from fake samples generated by G. Thus D returns a value D(x) ∈ which can be interpreted as the probability that the input sample (x) is a real sample from the data distribution. In this configuration, G is trained to obstruct D by generating samples which better resemble the real training samples, while D is continuously trained to tell apart real from fake samples. Crucially, G has no direct access to real samples from the training set, as it learns solely through its interaction with D. Both D and G are implemented by deep differentiable networks, typically consisting of multiple convolutional and fully-connected layers. They may be alternately trained using Stochastic Gradient Descent. In the short period of time since the introduction of the GAN model, many different enhancement methods and training variants have been suggested to improve their performance (see brief review below). Despite these efforts, often a large proportion of the generated samples is, arguably, not satisfactorily realistic. In some cases the generated sample does not resemble any of the real samples from the training set, and human observers find it difficult to classify synthetically generated samples to one of the classes which compose the training set (see illustration in FIG0).Figure 1: Images generated by different GANs trained on MNIST (top row), CelebA (middle row) and STL-10 (bottom row). Red square mark images of, arguably, low quality (best seen in color).This problem worsens with the increased complexity of the training set, and specifically when the training set is characterized by large inter-class and intra-class diversity. In this work we focus on this problem, aiming to improve the performance of GANs when the training dataset has large inter-class and intra-class diversity. Related Work. In an attempt to improve the performance of the original GAN model, many variants and extensions have been proposed in the past few years. These include architectural changes to G and D as in BID26, modifications to the loss function as in BID20; BID7, or the introduction of supervision into the training setting as in BID22; BID24. Another branch of related work, which is perhaps more closely related to our work, involves the learning of a meaningfully structured latent space. Thus Info-GAN decomposes the input noise into an incompressible source and a "latent code", Adversarial Auto-Encoders BID19 employ GANs to perform variational inference, and BID16 combine a Variational Auto-Encoder with a Generative Adversarial Network (see Appendix A for a more comprehensive description).Our Approach. Although modifications to the structure of the latent space have been investigated before as described above, the significance of the probability distribution used for sampling latent vectors was rarely investigated. A common practice today is to use a standard normal (e.g. N (0, I)) or uniform (e.g. U) probability distribution when sampling latent vectors from the latent space. We wish to challenge this common practice, and investigate the beneficial effects of modifying the distribution used to sample latent vectors in accordance with properties of the target dataset. Specifically, many datasets, especially those of natural images, are quite diverse, with high interclass and intra-class variability. At the same time, the representations of these datasets usually span high dimensional spaces, which naturally makes them very sparse. Intuitively, this implies that the underlying data distribution, which we try to learn using a GAN, is also sparse, i.e. it mostly consists of low-density areas with relatively few areas of high-density. We propose to incorporate this prior-knowledge into the model, by sampling latent vectors using a multi-modal probability distribution which better matches these characteristics of the data space distribution. It is important to emphasize that this architectural modification is orthogonal to, and can be used in conjunction with, other architectural improvements such as those reviewed above (see for instance FIG6 in Appendix D.) Supervision can be incorporated into this model by adding correspondence (not necessarily injective) between labels and mixture components. The rest of this paper is organized as follows: In Section 2 we describe the family of GM-GAN models. In Section 3 we offer an alternative method which focuses on measuring the trade-off between sample quality and diversity of generative models. In Section 4 we empirically evaluate our proposed model using various diverse datasets, showing that GM-GANs outperform the corresponding baseline methods with uni-modal distribution in the latent space. In Section 5 we describe a method for clustering datasets using GM-GANs, and provide qualitative and quantitative evaluation using various datasets of real images. Unsupervised GM-GAN. The target function which we usually optimize for, when training a GAN composed of Generator G and Discriminator D, can be written as follows: min DISPLAYFORM0 Above p X denotes the distribution of real training samples, and p Z denotes some d-dimensional prior distribution which is used as a source of stochasticity for the Generator. The corresponding loss functions of G and D can be written as follows: DISPLAYFORM1 DISPLAYFORM2 Usually, a multivariate uniform distribution (e.g. DISPLAYFORM3, or a multivariate normal distribution (e.g. N (0, I d×d)), are used as substitute for p Z when training GANs. In our proposed model, we optimize the same target function as in, but instead of using a unimodal random distribution for the prior p Z, we propose to use a multi-modal distribution which can better suit the inherent multi-modality of the real training data distribution, p X.Specifically, we propose to use a mixture of Gaussians as a multi-modal prior distribution where DISPLAYFORM4 Here K denotes the number of Gaussians in the mixture, DISPLAYFORM5 denotes a categorical random variable, and p k (z) denotes the multivariate Normal distribution N (µ k, Σ k), defined by the mean vector µ k, and the covariance matrix Σ k. In the absence of prior knowledge we assume a uniform mixture of Gaussians, that is, DISPLAYFORM6 Gaussian in the mixture can be fixed, or learned along with the parameters of the GAN in an "end-to-end" fashion to allow for a more flexible model. We investigated two corresponding variants of the new model -one (Static) where the the parameters of the Gaussians mixture are fixed throughout the model's training process, and one (Dynamic) where these parameters are allowed to change during the training process in order to potentially converge to a better solution. The details of these two variants are given in Appendix B.Supervised GM-GAN. In the supervised setting, we change the GM-GAN's discriminator so that instead of returning a single scalar, it returns a vector o ∈ R N where N is the number of classes in the dataset. Each element o i in this vector lies in. The Generator's purpose in this setting is, given a latent vector z sampled from the k'th Gaussian in the mixture, to generate a sample which will be classified by the discriminator as a real sample from class f (k), where f: DISPLAYFORM7 is a discrete function mapping identity of Gaussians to class labels. When K = N, f is bijective and the model is trained to map each Gaussian to a unique class in the data space. When K > N f is surjective, and multiple Gaussians can be mapped to the same class in order to model high diversity classes. When K < N f is injective, and multiple classes can be grouped together by mapping to the same Gaussian. We modify both loss functions of G and D to accommodate the class labels. The modified loss functions become the following: DISPLAYFORM8 where y(x) denotes the class label of sample x, and y(z) denotes the index of the Gaussian from which the latent vector z has been sampled. The training procedure for GM-GANs is fully described in Algorithm 1.Algorithm 1 Training the GM-GAN model. K -the number of Gaussians in the mixture. d -the dimension of the latent space (Z). c -the range from which the Gaussians' means are sampled. σ -scaling factor for the covariance matrices. iters -the number of training iterations. b D -the batch size for training the discriminator. b G -the batch size for training the Generator. γ -the learning rate. f -a mapping from Gaussian indices to class indices (in a supervised setting only). DISPLAYFORM0 init the mean vector of Gaussian k 3: DISPLAYFORM1 init the covariance matrix of Gaussian k 4: for i = 1...iters do 5: DISPLAYFORM2 Sample x j ∼ p X get a real sample from the training-set.7: DISPLAYFORM3 generate a fake sample using the Generator 10:if supervised then compute the loss of D 11: DISPLAYFORM4 else 14: DISPLAYFORM5 15: DISPLAYFORM6 16: DISPLAYFORM7 17: DISPLAYFORM8 update the weights of D by a single GD step.18: DISPLAYFORM9 generate a fake sample using the Generator if supervised then compute the loss of G 23: DISPLAYFORM0 else 25: DISPLAYFORM1 27: DISPLAYFORM2 update the weights of G by a single GD step. We describe next a new scoring method for GANs, which is arguably better suited for the task than the commonly used Inception Score proposed in BID28 and described in Appendix C. The Inception score has been used extensively over the last few years, but it has a number of drawbacks: (i) It is limited to the evaluation of GANS which are trained to generate natural images.(ii) It only measures the samples' inter-class diversity, ignoring the intra-class diversity of samples. (iii) It combines together a measure of quality and a measure of diversity into a single score. (iv) Different scores can be achieved by the same GAN, when sampling latent vectors with different parameters of the source probability distribution (e.g. σ, see Figure 6 in Appendix C).The Quality-Diversity trade-off: The quality of sample x ∈ X may be measured by its probability p X (x), which implies that samples drawn from dense areas in the source domain (i.e. close to the modes of the distribution) are mapped to high quality samples in the target domain, and vice versa. Therefore, we can increase the expected quality of generated samples in the target domain by sampling with high probability from dense areas of the source domain, and with low probability from sparse areas of the source domain. While increasing the expected quality of generated samples, this procedure also reduces the sample diversity 1. This fundamental trade-off between quality and diversity must be quantified if we want to compare the performance of different GAN models. Next we propose a new scoring method for either supervised or unsupervised GAN models, which is useful for multi-class image datasets, evaluating the trade-off between samples' quality and diversity. This scoring method also relies on a pre-trained classifier C, but unlike the Inception Score, this classifier is trained on the same training set on which the GAN is trained on. Classifier C is used to measure both the quality and the diversity of generated samples, as explained below. Quality Score To measure the quality of generated sample x, we propose to use the intermediate representation of x in the pre-trained classifier C, and to measure the Euclidean distance from this representation to its nearest-neighbor in the training set. More specifically, if C l (x) denotes the activation levels in the pre-trained classifier's layer l given sample x, then the quality score q(x), for sample x and a set of samples X, is defined as follows: DISPLAYFORM0 Above N N (x) denotes the nearest-neighbor of x in the training set, defined as N N (x) = arg min DISPLAYFORM1, and a denotes a positive constant. Diversity Score To measure the diversity of generated samples, we take into account both the inter-class, and the intra-class diversity. We measure intra-class diversity by the average (negative) MS-SSIM metric BID32 between all pairs of generated images in a given set of generated images X: DISPLAYFORM2 For inter-class diversity, we use the pre-trained classifier to classify the set of generated images, such that for each sampled image x, we have a classification prediction in the form of a one-hot vector c(x). We then measure the entropy of the average one-hot classification prediction vector to evaluate the diversity between classes in the samples set: DISPLAYFORM3 Finally, the diversity score is defined as the geometric mean of FORMULA24 and FORMULA25: DISPLAYFORM4 4 EMPIRICAL EVALUATION In this section we empirically evaluate the benefits of our proposed approach, comparing the performance of GM-GAN with the corresponding baselines. Thus we compare the performance of the unsupervised GM-GAN model to that of the originally proposed GAN BID6, and the performance of our proposed supervised GM-GAN model to that of AC-GAN BID24. In both cases, the baseline models' latent space probability distribution is standard normal, i.e. z ∼ N (0, I). The network architectures and hyper-parameters used for training the GM-GAN models are similar to those used for training the baseline models. For the most part we used the Static GM-GAN with default values d = 100, c = 0.1, σ = 0.15, B D = 64, b G = 128, γ = 0.0002; K and iters varied in the different experiments. The Dynamic GM-GAN model was only used in the experiments summarized in FIG3.In the following experiments we evaluate the different models on the 6 datasets listed in We first compare the performance of our proposed GM-GAN models to the aforementioned baseline models using a toy dataset, which has been created in order to gain more intuition regarding the properties of the GM-GAN model. The dataset consists of 5,000 training samples, where each training sample x is a point in R 2 drawn from a homogeneous mixture of K Gaussians, i.e., ∀x DISPLAYFORM0 In our experiments we used K = 9 Gaussians, ∀k ∈ [K] Σ k = 0.1 * I and µ = {−1, 0, 1} × {−1, 0, 1}. We labeled each sample with the identity of the Gaussian from which it was sampled. We trained two instances of the GM-GAN model, one supervised using the labels of the samples, and one unsupervised oblivious of these labels. In both cases, we used K = 9 Gaussians in the mixture from which latent vectors were sampled. Figure 2 presents samples generated by the baseline models (GAN, AC-GAN) and samples generated by our proposed GM-GAN models (both unsupervised and supervised variants). It is clear that both variants of the GM-GAN generate samples with a higher likelihood, which matches the original distribution more closely as compared to the baseline methods. The trade-off between quality and diversity is illustrated in Figure 3, showing high quality and low diversity for σ = 0.25, and vice versa for σ = 2.0 (see Section 4.3). An intriguing observation is that the GM-GAN's Generator is capable, without any supervision, of mapping each Gaussian in the latent space to samples in the data-space which are almost perfectly aligned with a single Gaussian. We also observe this phenomenon when training unsupervised GM-GAN on the MNIST and Fashion-MNIST datasets. In Section 5 we exploit this phenomenon to achieve a clustering algorithm. Finally, we note that the GM-GAN models converge considerably faster than the classical GAN model, see FIG4 in Appendix D. We next turn to evaluate our proposed models when trained on more complex datasets. We start by using the customary Inception Score BID28 to evaluate and compare the performance of the difference models, the two GM-GAN models and the baseline models (GAN and AC-GAN). We trained the models on two real datasets with 10 classes each, CIFAR-10 and STL-10 (see TAB0). Each variant of the GM-GAN model was trained multiple times, each time using a different number (K) of Gaussians in the latent space probability distribution. In addition, each model was trained 10 times using different initial parameter values. We then computed for each model its mean Inception Score and the corresponding standard error. The for the two unsupervised and two supervised models are presented in TAB2. In all cases, the two GM-GAN models achieve higher scores when compared to the respective baseline model. The biggest improvement is achieved in the supervised case, where the supervised variant of the GM-GAN model outperforms AC-GAN by a large margin. Model (unsupervised) Score As discussed in Section 3, the Inception Score is not sufficient, on its own, to illustrate the trade-off between the quality and the diversity of samples which a certain GAN is capable of generating. In our experiments, we control the quality-diversity trade-off by varying, after the model's training, the probability distribution which is used to sample latent vectors from the latent space. We do so by multiplying the covariance matrix of each Gaussian by a scaling factor σ. Specifically, when using the baseline models we sample z ∼ N (0, σ * I), and when using the GM-GAN models we DISPLAYFORM0 Thus, when σ < 1, latent vectors are sampled with lower variance around the modes of the latent space probability distribution, and therefore the respective samples generated by the Generator are of higher expected quality, but lower expected diversity. The opposite happens when σ > 1, where the respective samples generated by the Generator are of lower expected quality, but higher expected diversity. Figures 3, 4 demonstrate qualitatively the quality-diversity trade-off offered by GM-GANs when trained on the Toy and MNIST datasets. DISPLAYFORM1 Figure 4: Samples taken from a GM-GAN trained on the MNIST dataset. In each panel, latent vectors samples are drawn using different σ values (σ = 1.0 was used during training). Clearly The quality of samples decreases, and the diversity increases, as σ grows. Next, we evaluated each model by calculating our proposed Quality Score from Eq. FORMULA22, and the Combined Diversity Score from Eq. FORMULA26, for each σ ∈ {0.5, 0.6, ..., 1.9, 2.0}. Each model was trained 10 times using different initial parameter values. We computed for each model its mean Quality and mean Combined Diversity scores and the corresponding standard errors. The Quality and Diversity Scores of the GM-GAN and baseline models, when trained on the CIFAR-10 and STL-10 datasets, are presented in FIG3 (see additional datasets in FIG5 in Appendix D).In some cases (e.g. supervised training on CIFAR-10 and STL-10) the show a clear advantage for our proposed model as compared to the baseline, as both the quality and the diversity scores of GM-GAN surpass those of AC-GAN, for all values of σ. In other cases (e.g. unsupervised training on CIFAR-10 and STL-10), the show that for the lower-end range of σ, the baseline model generates images of higher quality but dramatically lower diversity, as compared to our proposed model. In accordance, when visually examining the samples generated by the two models, we notice that most samples generated by the baseline model belong to a single class, while samples generated by our model are much more diverse and are scattered uniformly among the different classes. In all cases, the charts predictably show an ascending Quality Score, and a descending Combined Diversity Score, as σ is increased. Throughout our experiments, we noticed an intriguing phenomenon where the unsupervised variant of GM-GAN tends to map latent vectors sampled from different Gaussians in the latent space to samples of different classes in the data space. Specifically, each Gaussian in the latent space is usually mapped, by the GM-GAN's Generator, to a single class in the data space. FIG0 in Appendix D demonstrates this phenomenon using different datasets. The fact that the latent space in our proposed model is sparse, while being composed of multiple Gaussians with little overlap, may be the underlying reason for this phenomenon. In this section we exploit this observation to develop a new clustering algorithm, and provide quantitative evaluation of the proposed method. Clustering Method In the proposed method we first train an unsupervised GM-GAN which receives as input the data points without corresponding labels. K, the number of Gaussians forming the latent space, is set to equal the number of clusters in the intended partition. Using the trained GM-GAN model, we sample from each Gaussian k ∈ [K] a set of M latent vectors, from which we generate a set of M synthetic samples DISPLAYFORM0. We then train a K-way multi-class classifier on the unified set of samples from all Gaussians k∈[K] X k, where the label of sample x ∈ X k is set to k, i.e. the index of the Gaussian from which the corresponding latent vector has been sampled. Finally, we obtain the soft-assignment to clusters of each sample x in the original dataset by using the output of this classifier c(x) ∈ K when given x as input. Each element c(x) k (k ∈ [K]) of this output vector marks the association level of the sample x to the cluster k. Hard-assignment to clusters can be trivially obtained from the soft-assignment vector by selecting the most likely clusterk = arg max k∈ [K] c(x) k. This procedure is formally described in Algorithm 2.Algorithm 2 Unsupervised clustering procedure using GM-GANs. Require:X -a set of samples to cluster. K -number of clusters. M -number of samples to draw from each Gaussian. DISPLAYFORM1 Train an unsupervised GM-GAN on X using K Gaussians. DISPLAYFORM2 Sample M latent vectors from the k'th latent Gaussian. Generate M samples using the set of latent vectors Z k. ∀ x ∈ X k y(x) ← k Label every sample by the Gaussian from which it was generated. 6: X ← k X k Unite all samples into the set X. 7: c ← classifier(X, y)Train a classifier on samples X and labels y. DISPLAYFORM0 Cluster X using classifier c. Method ACC NMI MNIST K-Means 0.5349 0.500 AE + K-Means 0.8184 -DEC 0.8430 -DCEC BID8 0.8897 0.8849 InfoGAN 0.9500 -CAE-l 2 + K-Means BID1 0.9511 -CatGAN 0.9573 -DEPICT BID4 0.9650 0.9170 DAC BID2 0.9775 0.9351 GAR BID11 0.9832 -IMSAT BID9 0 Table 3: Clustering performance of our method on different datasets. Scores are based on clustering accuracy (ACC) and normalized mutual information (NMI). Results of a broad range of recent existing solutions are also presented for comparison. The of alternative methods are the ones reported by the authors in the original papers. Methods marked with (*) are based on our own implementation, as we didn't find any published scores to compare to. We evaluated the proposed clustering method on three different datasets: MNIST, Fashion-MNIST, and a subset of the Synthetic Traffic Signs Dataset containing 10 selected classes (see TAB0). To evaluate clustering performance we adopt two commonly used metrics: Normalized Mutual Information (NMI), and Clustering Accuracy (ACC). Clustering accuracy measures the accuracy of the hard-assignment to clusters, with respect to the best permutation of the dataset's ground-truth labels. Normalized Mutual Information measures the mutual information between the ground-truth labels and the predicted labels based on the clustering method. The range of both metrics is. The unsupervised clustering scores of our method are presented in Table 3. We noet in passing that Algorithm 2 can be implemented with other GAN variants which are augments with a GM distribution of the latent space with similar beneficial , see FIG6 in Appendix D. This work is motivated by the observation that the commonly used GAN architecture may be ill suited to model data in such cases where the training set is characterized by large inter-class and intra-class diversity, a common case with real-world datasets these days. To address this problem we propose a variant of the basic GAN model where the probability distribution over the latent space is a mixture of Gaussians, a multi-modal distribution much like the target data distribution which the GAN is trained to model. This model can be used with or without label supervision. In addition, the proposed modifications can be applied to any GAN model, regardless of the specifics of the loss function and architecture (see, for example, FIG6 in Appendix D.).In our empirical study, using both synthetic and real-world datasets, we quantitatively showed that GM-GANs outperform baselines, both when evaluated using the commonly used Inception Score BID28, and when evaluated using our own alternative scoring method. We also demonstrated how the quality-diversity trade-off offered by our models can be controlled by altering, post-training, the probability distribution of the latent space. This allows one to sample higherquality, lower-diversity samples or vice versa, according to one's needs. Finally, we qualitatively demonstrated how the unsupervised variant of GM-GAN tends to map latent vectors sampled from different Gaussians in the latent space to samples of different classes in the data space. We further showed how this phenomenon can be exploited for the task of unsupervised clustering, and backed our method with quantitative evaluation. GANs have been extensively used in the domain of computer-vision, where their applications include super resolution from a single image BID18 ), text-to-image translation BID27, image-to-image translation BID10 BID12, image in-painting BID36 and video completion BID21. Aside from their use in the computer-vision domain, GANs have been used for other tasks such as semi-supervised learning BID14, music generation BID5 ), text generation BID37 and speech enhancement BID25.Subsequently much effort was directed at improving GANs through architectural changes to G and D, as in the DCGANs described in BID26. Improved performance was reported in BID20; BID7, among others, by modifying the loss function used to train the GAN model. Additional improvement was achieved by introducing supervision into the training setting, as in conditional GANs BID22 BID24. These conditional variants were shown to enhance the quality of the generated sample, while also improving the stability of the notorious training process of these models. In an effort to impose a meaningfully structure on the latent space, Info-GAN decomposes the input noise into an incompressible source and a "latent code", attempting to discover latent sources of variation by maximizing the mutual information between the latent code and the Generator's output. This latent code can be used to discover object classes in a purely unsupervised fashion, although it is not strictly necessary that the latent code be categorical. Adversarial AutoEncoders BID19 employ GANs to perform variational inference by matching the aggregated posterior of the auto-encoder's hidden latent vector with an arbitrary prior distribution. As a , the decoder of the adversarial auto-encoder learns a deep generative model that maps the imposed prior to the data distribution. BID16 combined a Variational Auto-Encoder with a Generative Adversarial Network in order to use the learned feature representations in the GAN's discriminator as basis for the VAE reconstruction objective. As a , this hybrid model is capable of learning a latent space in which high-level abstract visual features (e.g. wearing glasses) can be modified using simple arithmetic of latent vectors. The parameters of the mixture of Gauusian distribution used to sample the latent vector can be fixed or learned. One may be able to choose these parameters by using prior knowledge, or pick them randomly. Perhaps a more robust solution is to learn the parameters of the Gaussian Mixture along with the parameters of the GAN in an "end-to-end" fashion. This should, intuitively, allow for a more flexible, and perhaps better performing model. We therefore investigated two variants of the new model -one (static) where the the parameters of the Gaussians mixture are fixed throughout the model's training process, and one (dynamic) where these parameters are allowed to change during the training process in order to potentially converge to a better a solution. These variants are described in detail next:Static GM-GAN. In the basic GM-GAN model, which we call Static Multi-Modal GAN (Static GM-GAN), we assume that the parameters of the mixture of Gaussians distribution are fixed before training the model, and cannot change during the training process. More specifically, each of the mean vectors µ k is uniformly sampled from the multivariate uniform distribution U [−c, c] d, and each of the covariance matrices Σ k has the form of σ * I d×d, where c ∈ R and σ ∈ R are hyper-parameters left to be determined by the user. Dynamic GM-GAN. We extend our basic model in order to allow for the dynamic tuning of parameters for each of the Gaussians in the mixture. We start by initializing the mean vectors and covariance matrices as in the static case, but we include them in the set of learnable parameters that are optimized during the GAN's training process. This modification allows the Gaussians' means to wander to new locations, and lets each Gaussian have a unique covariance matrix. This potentially allows the model to converge to a better local optimum, and achieve better performance. The architecture of the Dynamic GM-GAN is modified so that G receives as input a categorical random variable k, which determines from which Gaussian the latent vector should be sampled. This vector is fed into a stochastic node used for sampling latent vectors given the Gaussian's index, i.e. z|k ∼ N (µ k, Σ k). In order to optimize the parameters of each Gaussian in the training phase, back-propagation would have to be performed through this stochastic node, which is not possible. To overcome this obstacle, we use the re-parameterization trick as suggested by BID13: instead of sampling z ∼ N (µ k, Σ k) we sample ∼ N (0, I) and define z = A k + µ k, where A ∈ R d×d and µ k ∈ R d are parameters of the model, and d is the dimension of the latent space. We thus get µ(z) = µ k and Σ(z) = A k A (a) (b) Figure 6: Inception Scores of Static GM-GAN models trained on (a) CIFAR-10 and (b) STL-10, when latent vectors are sampled using different values of σ. In both cases, the same model achieves very different Inception Scores when different values of σ are used. Both models were trained using σ = 1. Note that the best score is obtained for σ < 1, far from the training value σ = 1. ) trained on the fashion-MNIST dataset. We evaluated the original model with uniform and Gaussian latent space distribution. In addition, we incorporated multi-modal Gaussian distribution of the latent space into the model with 3, 5 and 10 mixture components. The of the GM-info-GAN variants are clearly better than vanilla, with best obtained for 5 Gaussian components. (a) (b) FIG0: Samples taken from two unsupervised GM-GAN models trained on the MNIST (top panels), Fashion-MNIST (middle panels) and CIFAR-10 (bottom panels) datasets. In (a) the Gaussian mixture contains K = 10 Gaussians; in each panel, each row contains images sampled from a different Gaussian. In (b) the Gaussian mixture contains K = 20 Gaussians; in each panel, each half row contains images sampled from a different Gaussian.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Multi modal Guassian distribution of latent space in GAN models improves performance and allows to trade-off quality vs. diversity

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.


Question: Represent the article for finding a single-sentence TLDR

Question: Distributed optimization is essential for training large models on large datasets. Multiple approaches have been proposed to reduce the communication overhead in distributed training, such as synchronizing only after performing multiple local SGD steps, and decentralized methods (e.g., using gossip algorithms) to decouple communications among workers. Although these methods run faster than AllReduce-based methods, which use blocking communication before every update, the ing models may be less accurate after the same number of updates. Inspired by the BMUF method of , we propose a slow momentum (SloMo) framework, where workers periodically synchronize and perform a momentum update, after multiple iterations of a base optimization algorithm. Experiments on image classification and machine translation tasks demonstrate that SloMo consistently yields improvements in optimization and generalization performance relative to the base optimizer, even when the additional overhead is amortized over many updates so that the SloMo runtime is on par with that of the base optimizer. We provide theoretical convergence guarantees showing that SloMo converges to a stationary point of smooth non-convex losses. Since BMUF is a particular instance of the SloMo framework, our also correspond to the first theoretical convergence guarantees for BMUF.

SlowMo improves the optimization and generalization performance of communication-efficient decentralized algorithms without sacrificing speed.

Represent the TLDR for finding the corresponding scientific article

We show that deep learning network derivatives have a low-rank structure, and this structure allows us to use second-order derivative information to calculate learning rates adaptively and in a computationally feasible manner.


Question: Represent the scientific article to find a short TLDR

Question: Structural planning is important for producing long sentences, which is a missing part in current language generation models. In this work, we add a planning phase in neural machine translation to control the coarse structure of output sentences. The model first generates some planner codes, then predicts real output words conditioned on them. The codes are learned to capture the coarse structure of the target sentence. In order to learn the codes, we design an end-to-end neural network with a discretization bottleneck, which predicts the simplified part-of-speech tags of target sentences. Experiments show that the translation performance are generally improved by planning ahead. We also find that translations with different structures can be obtained by manipulating the planner codes. When human speaks, it is difficult to ensure the grammatical or logical correctness without any form of planning. Linguists have found evidence through speech errors or particular behaviors that indicate speakers are planning ahead BID16. Such planning can happen in discourse or sentence level, and sometimes we may notice it through inner speech. In contrast to human, a neural machine translation (NMT) model does not have the planning phase when it is asked to generate a sentence. Although we can argue that the planning is done in the hidden layers, however, such structural information remains uncertain in the continuous vectors until the concrete words are sampled. In tasks such as machine translation, a source sentence can have multiple valid translations with different syntactic structures. As a consequence, in each step of generation, the model is unaware of the "big picture" of the sentence to produce, ing in uncertainty of word prediction. In this research, we try to let the model plan the coarse structure of the output sentence before decoding real words. As illustrated in FIG0, in our proposed framework, we insert some planner codes into the beginning of the output sentences. The sentence structure of the translation is governed by the codes. An NMT model takes an input sentence X and produce a translation Y. Let S Y denotes the syntactic structure of the translation. Indeed, the input sentence already provides rich information about the target-side structure S Y.For example, given the Spanish sentence in FIG0, we can easily know that the translation will have a noun, a pronoun and a verb. Such obvious structural information does not have uncertainty, and thus does not require planning. In this example, the uncertain part is the order of the noun and the pronoun. Thus, we want to learn a set of planner codes C Y to disambiguate such uncertain information about the sentence structure. By conditioning on the codes, we can potentially increase the effectiveness of beam search as the search space is properly regulated. In this work, we use simplified POS tags to annotate the structure S Y. We learn the planner codes by putting a discretization bottleneck in an end-to-end network that reconstructs S Y with both X and C Y. The codes are merged with the target sentences in the training data. Thus, no modification to the NMT model is required. Experiments show the translation performance is generally improved with structural planning. More interestingly, we can control the structure of output sentences by manipulating the planner codes. In this section, we first extract the structural annotation S Y by simplifying the POS tags. Then we explain the code learning model for obtaining the planner codes. To reduce uncertainty in the decoding phase, we want a structural annotation that describes the "big picture" of the sentence. For instance, the annotation can tell whether the sentence to generate is in a "NP VP" order. The uncertainty of local structures can be efficiently solved by beam search or the NMT model itself. In this work, we extract such coarse structural annotations S Y through a simple two-step process that simplifies the POS tags of the target sentence:1. Remove all tags other than "N", "V", "PRP", "," and ".". Note that all tags begin with "N" (e.g. NNS) are mapped to "N", and tags begin with "V" (e.g. VBD) are mapped to "V". The following list gives an example of the process:Input: He found a fox behind the wall. Step 1: PRP V N N.Step 2: PRP V N.Note that many other annotations can also be considered to represent the syntactic structure, which is left for future work to explore. Next, we learn the planner codes C Y to remove the uncertainty of the sentence structure S Y when producing a translation. For simplicity, we use the notion S and C to replace S Y and C Y in this section. DISPLAYFORM0 Architecture of the code learning model. The discretization bottleneck is shown as the dashed lines. We first compute the discrete codes C 1,.., C N based on simplified POS tags S 1,..., S T: DISPLAYFORM1 DISPLAYFORM2 where the tag sequence S 1,..., S T is firstly encoded using a backward LSTM BID4. E(·) denotes the embedding function. Then, we compute a set of vectorsC 1,...,C N, which are latterly discretized in to approximated one-hot vectors C 1,..., C N using Gumbel-Softmax trick BID5 BID10. We then combine the information from X and C to initialize a decoder LSTM that sequentially predicts S 1,..., S T: DISPLAYFORM3 DISPLAYFORM4 where [C 1, ..., C N] denotes a concatenation of N one-hot vectors. Note that only h t is computed with a forward LSTM. Both f enc and f dec are affine transformations. Finally, we predict the probability of emitting each tag S t with DISPLAYFORM5 The architecture of the code learning model is depicted in Fig. 2, which can be seen as a sequence auto-encoder with an extra context input X to the decoder. The parameters are optimized with crossentropy loss. Once the code learning model is trained, we can obtain the planner codes C for all target sentences in the training data using the encoder part. The training data of machine translation dataset is composed of (X, Y) sentence pairs. With the planner codes C Y we obtained, our training data now becomes a list of (X, C Y ; Y) pairs. As shown in FIG0, we connect the planner codes and target sentence with a " eoc " token. With the modified dataset, we train a regular NMT model. We use beam search when decoding sentences, thus the planner codes are searched before emitting real words. The codes are removed from the translation during evaluation. Recently, some methods are proposed to improve the syntactic correctness of the translations. BID19 restricts the search space of the NMT decoder using the lattice produced by a Statistical Machine Translation system. BID2 takes a multi-task approach, letting the NMT model to parse a dependency tree and combine the parsing loss with the original loss. Several works further incorporate the targetside syntactic structures explicitly. BID12 interleaves CCG supertags with normal output words in the target side. Instead of predicting words, trains a NMT model to generate linearized constituent parse trees. BID20 proposed a model to generate words and parse actions simultaneously. The word prediction and action prediction are conditioned on each other. However, none of the these methods plan the structure before translation. Similar to our code learning approach, some works also learn the discrete codes for different purposes. compresses the word embeddings by learning the concept codes to represent each word. BID7 breaks down the dependency among words with shorter code sequences. The decoding can be faster by predicting the shorter artificial codes. We evaluate our models on IWSLT 2014 Germanto-English task BID1 and ASPEC Japanese-to-English task BID13, containing 178K and 3M bilingual pairs respectively. We use Kytea BID15 to tokenize Japanese texts and moses toolkit BID8 for other languages. Using bytepair encoding BID17 In the code learning model, all hidden layers have 256 hidden units. The model is trained using Nesterov's accelerated gradient (NAG) BID14 for maximum 50 epochs with a learning rate of 0.25. We test different settings of code length N and the number of code types K. The information capacity of the codes will be N log K bits. In TAB1, we evaluate the learned codes for different settings. S y accuracy evaluates the accuracy of correctly reconstructing S y with the source sentence X and the code C y. C y accuracy reflects the chance of guessing the correct code C y given X.We can see a clear trade-off between S Y accuracy and C Y accuracy. When the code has more capacity, it can recover S Y more accurately, however, ing in a lower probability for the NMT model to guess the correct code. We found the setting of N = 2, K = 4 has a balanced trade-off. To make a strong baseline, we use 2 layers of bidirectional LSTM encoders with 2 layers of LSTM decoders in the NMT model. The hidden layers have 256 units for IWSLT De-En task and 1000 units for ASPEC Ja-En task. We apply Key-Value Attention in the first decoder layer. Residual connection BID3 ) is used to combine the hidden states in two decoder layers. Dropout is applied everywhere outside of the recurrent function with a drop rate of 0.2. To train the NMT models, we also use the NAG optimizer Model BLEU(%) BS=1 BS=3 BS=5 with a learning rate of 0.25, which is annealed by a factor of 10 if no improvement of loss value is observed in 20K iterations. Best parameters are chosen on a validation set. As shown in TAB3, by conditioning the word prediction on the generated planner codes, the translation performance is generally improved over a strong baseline. The improvement may be the of properly regulating the search space. However, when we apply greedy search on JaEn dataset, the BLEU score is much lower compared to the baseline. We also tried to beam search the planner codes then switch to greedy search, but the are not significantly changed. We hypothesize that it is important to simultaneously explore multiple candidates with drastically different structures on Ja-En task. By planning ahead, more diverse candidates can be explored, which improves beam search but not greedy search. If so, the are in line with a recent study BID9 that shows the performance of beam search depends on the diversity of candidates. Instead of letting the beam search to decide the planner codes, we can also choose the codes manually. Table 3 gives an example of the candidate translations produced by the model when conditioning on different planner codes.input AP no katei ni tsuite nobeta. (Japanese) code 1 <c4> <c1> <eoc> the process of AP is described.code 2 <c1> <c1> <eoc> this paper describes the process of AP.code 3 <c3> <c1> <eoc> here was described on process of AP.code 4 <c2> <c1> <eoc> they described the process of AP. Table 3: Example of translation conditioned on different planner codes in Ja-En task <c1> <c1> <c1> <c2> <c1> <c3> <c1> <c4> <c2> <c1> <c2> <c2> <c2> <c3> <c2> <c4> <c3> <c1> <c3> <c2> <c3> <c3> <c3> <c4> <c4> <c1> <c4> <c2> <c4> <c3> <c4> <c4> 4% 8% 12% Figure 3: Distribution of assigned planner codes for English sentences in ASPEC Ja-En datasetAs shown in Table 3, we can obtain translations with drastically different structures by manipulating the codes. The show that the proposed method can be useful for sampling paraphrased translations with high diversity. The distribution of the codes learned for 3M English sentences in ASPEC Ja-En dataset is shown in Fig. 3. We found the code "<c1> <c1>" is assigned to 20% of the sentences, whereas "<c4> <c3>" is not assigned to any sentence. The skewed distribution may indicate that the capacity of the codes is not fully exploited, and thus leaves room for further improvement. Instead of learning discrete codes, we can also directly predict the structural annotations (e.g. POS tags), then translate based on the predicted structure. However, as the simplified POS tags are also long sequences, the error of predicting the tags will be propagated to word generation. In our experiments, doing so degrades the performance by around 8 BLEU points on IWSLT dataset. In this paper, we add a planning phase in neural machine translation, which generates some planner codes to control the structure of the output sentence. To learn the codes, we design an end-to-end neural network with a discretization bottleneck to predict the simplified POS tags of target sentences. Experiments show that the proposed method generally improves the translation performance. We also confirm the effect of the planner codes, by being able to sample translations with drastically different structures using different planner codes. The planning phase helps the decoding algorithm by removing the uncertainty of the sentence structure. The framework described in this paper can be extended to plan other latent factors, such as the sentiment or topic of the sentence.

represent the tldr for finding the corresponding scientific article

Plan the syntactic structural of translation using codes

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

We provide insightful understanding of sequence-labeling NER and propose to use two types of cross structures, both of which bring theoretical and empirical improvements.