Question: represent the article for finding a single-sentence tldr:

Question: As distributed approaches to natural language semantics have developed and diversified, embedders for linguistic units larger than words (e.g., sentences) have come to play an increasingly important role. To date, such embedders have been evaluated using benchmark tasks (e.g., GLUE) and linguistic probes. We propose a comparative approach, nearest neighbor overlap (N2O), that quantifies similarity between embedders in a task-agnostic manner. N2O requires only a collection of examples and is simple to understand: two embedders are more similar if, for the same set of inputs, there is greater overlap between the inputs' nearest neighbors. We use N2O to compare 21 sentence embedders and show the effects of different design choices and architectures. Continuous embeddings-of words and of larger linguistic units-are now ubiquitious in NLP. The success of self-supervised pretraining methods that deliver embeddings from raw corpora has led to a proliferation of embedding methods, with an eye toward "universality" across NLP tasks. Our focus here is on sentence embedders, and specifically their evaluation. As with most NLP components, intrinsic (e.g., and extrinsic (e.g., GLUE;) evaluations have emerged for sentence embedders. Our approach, nearest neighbor overlap (N2O), is different: it compares a pair of embedders in a linguistics-and task-agnostic manner, using only a large unannotated corpus. The central idea is that two embedders are more similar if, for a fixed query sentence, they tend to find nearest neighbor sets that overlap to a large degree. By drawing a random sample of queries from the corpus itself, N2O can be computed on in-domain data without additional annotation, and therefore can help inform embedder choices in applications such as text clustering , information retrieval , and open-domain question answering , among others. After motivating and explaining the N2O method (§2), we apply it to 21 sentence embedders (§3-4). Our findings (§5) reveal relatively high functional similarity among averaged static (noncontextual) word type embeddings, a strong effect of the use of subword information, and that BERT and GPT are distant outliers. In §6, we demonstrate the robustness of N2O across different query samples and probe sizes. We also illustrate additional analyses made possible by N2O: identifying embeddingspace neighbors of a query sentence that are stable across embedders, and those that are not (§7); and probing the abilities of embedders to find a known paraphrase (§8). The latter reveals considerable variance across embedders' ability to identify semantically similar sentences from a broader corpus. We first motivate and introduce our nearest neighbor overlap (N2O) procedure for comparing embedders (maps from objects to vectors). Although we experiment with sentence embedders in this paper, we note that this comparison procedure can be applied to other types of embedders (e.g., phrase-level or document-level). Computation of N2O for two embedders, e A and e B, using a corpus C; the number of nearest neighbors is given by k. n is the number of queries (q 1 . . . q n), which are sampled uniformly from the corpus without replacement. The output is in, where 0 indicates no overlap between nearest neighbors for all queries, and 1 indicates perfect overlap. We would like to quantify the extent to which sentence embedders vary in their treatment of "similarity." For example, given the sentence Mary gave the book to John, embedders based on bag-of-words will treat John gave the book to Mary as being maximally similar to the first sentence, whereas other embedders may treat Mary gave the dictionary to John as more similar; our comparison should reflect this intuition. We would also like to focus on using naturally-occuring text for our comparison. Although there is merit in expert-constructed examples (see linguistic probing tasks referenced in §9), we have little understanding of how these models will generalize to text from real documents; many application settings involve computing similarity across texts in a corpus. Finally, we would like our evaluation to be task-agnostic, since we expect embeddings learned from large unannotated corpora in a self-supervised (and task-agnostic) manner to continue to play an important role in NLP. As a , we base our comparison on nearest neighbors: first, because similarity is often assumed to correspond to nearness in embedding space (e.g., Figure 1); second, because nearest neighbor methods are used directly for retrieval and other applications; and finally, because the nearest neighbors of a sentence can be computed for any embedder on any corpus without additional annotation. Suppose we want to compare two sentence embedders, e A (·) and e B (·), where each embedding method takes as input a natural language sentence s and outputs a d-dimensional vector. For our purposes, we consider variants trained on different data or using different hyperparameters, even with the same parameter estimation procedure, to be different sentence embedders. Take a corpus C, which is likely to have some semantic overlap in its sentences, and segment it into sentences s 1,..., s |C|. Randomly select a small subset of the sentences in C as "queries" (q 1, . . ., q n). To see how similar e A and e B are, we compute the overlap in nearest neighbor sentences, averaged across multiple queries; the algorithm is in Figure 2. nearest(e i, q j, C, k) returns the k nearest neighbor sentences in corpus C to the query sentence q j, where all sentences are embedded with e i. 2 There are different ways to define nearness and distance in embedding spaces (e.g., using cosine similarity or Euclidean distance); in this paper we use cosine similarity. We can think about this procedure as randomly probing the sentence vector space (through the n query sentences) from the larger space of the embedded corpus, under a sentence embedder e i; in some sense, k controls the depth of the probe. The N2O procedure then compares the sets of sentences recovered by the probes. In the previous section, we noted that we consider a "sentence embedder" to encompass how it was trained, which data it was trained on, and any other hyperparameters involved in its creation. In this section, we first review the broader methods behind these embedders, turning to implementation decisions in §4. We consider tf-idf, which has been clasically used in information retrieval settings. The tf-idf of a word token is based off two statistics: term frequency (how often a term appears in a document) and inverse document frequency (how rare the term is across all documents). The vector representation of the document is the idf-scaled term frequencies of its words; in this work we treat each sentence as a "document" and the vocabulary-length tf-idf vector as its embedding. Because sentence embeddings are often built from word embeddings (through initialization when training or other composition functions), we briefly review notable word embedding methods. Static embeddings. We define "static embeddings" to be fixed representations of every word type in the vocabulary, regardless of its context. We consider three popular methods: word2vec embeddings optimized to be predictive of a word given its context (continuous bag of words) or vice versa (skipgram); GloVe embeddings learned based on global cooccurrence counts; and FastText , an extension of word2vec which includes character n-grams (for computing representations of out-of-vocabulary words). Contextual embeddings. Contextual word embeddings, where a word token's representation is dependent on its context, have become popular due to improvements over state-of-the-art on a wide variety of tasks. We consider: • ELMo embeddings are generated from a multi-layer, bidirectional recurrent language model that incorporates character-level information. • GPT embeddings are generated from a unidirectional language model with multi-layer transformer decoder; subword information is included via byte-pair encoding (BPE;). • BERT embeddings are generated from a transformer-based model trained to predict (a) a word given both left and right context, and (b) whether a sentence is the "next sentence" given a previous sentence. Subword information is incorporated using the WordPiece model . Composition of word embeddings. The simplest way to obtain a sentence's embedding from its sequence of words is to average the word embeddings. 3 Despite the fact that averaging discards word order, it performs surprisingly well on sentence similarity, NLI, and other downstream tasks . 4 In the case of contextual embeddings, there may be other conventions for obtaining the sentence embedding, such as using the embedding for a special token or position in the sequence. With BERT, the [CLS] token representation (normally used as input for classification) is also sometimes used as a sentence representation; similarly, the last token's representation may be used for GPT. Table 1: Details of the pretrained sentence embedders we test in this paper. For methods which produce word embeddings, "composition" denotes how a single embedding was obtained from the sentence's word embeddings. ELMo embeddings are averaged across the three bi-LSTM layers; BERT and GPT embeddings come from the final hidden layer. All of the models besides tf-idf and the fine-tuned version of BERT are common pretrained versions; further details are in Appendix A. A more direct way to obtain sentence embeddings is to learn an encoding function that takes in a sequence of tokens and outputs a single embedding; often this is trained using a relevant supervised task. We consider two encoder-based methods: • InferSent : supervised training on the Stanford Natural Language Inference (SNLI;) dataset; the sentence encoder provides representations for the premise and hypothesis sentences, which are then fed into a clasifier. • Universal Sentence Encoder (USE;): supervised, multi-task training on several semantic tasks (including semantic textual similarity); sentences are encoded either with a deep averaging network or a transformer. Our main experiment is a broad comparison, using N2O, of the embedders discussed above and listed in Table 1. Despite the vast differences in methods, N2O allows us to situate each in terms of its functional similarity to the others. N2O computation. We describe a N2O sample as, for a given random sample of n queries, the computation of N2O(e A, e B, C, k) for every pair of sentence embedders as described in §2, using cosine similarity to determine nearest neighbors. The in §5 are with k (the number of sentences retrieved) set to 50, averaged across five samples of n = 100 queries. We illustrate the effects of different k and N2O samples in §6. Corpus. For our corpus, we draw from the English Gigaword , which contains newswire text from seven news sources. For computational feasibility, we use the articles from 2010, for a total of approximately 8 million unique sentences. 5 We note preprocessing details (segmentation, tokenization) in Appendix A. Queries. For each N2O sample, we randomly select 100 ledes (opening sentences) from the news articles of our corpus, and use the same ones across all embedders. Because the Gigaword corpus contains text from multiple news sources covering events over the same time period, it is likely that the corpus will contain semantically similar sentences for a given lede. The average query length is 30.7 tokens (s.d. 10.2); an example query is: "Sandra Kiriasis and brakewoman Stephanie Schneider of Germany have won the World Cup bobsled race at Lake Placid." Sentence embedders. Table 1 details the sentence embedders we use in our experiments. In general, we use popular pretrained versions of the methods described in §3. We also select pretrained variations of the same method (e.g., FastText embeddings trained from different corpora) to permit more controlled comparisons. In a couple of cases, we train/finetune models of our own: for tf-idf, we compute frequency statistics using our corpus, with each sentence as its own "document"; for BERT, we use the Hugging Face implementation with default hyperparameters, 6 and finetune using the matched subset of MultiNLI for three epochs (dev. accuracy 84.1%). We note that additional embedders are easily situated among the ones tested in this paper by first computing nearest neighbors of the same query sentences, and then computing overlap with the nearest neighbors obtained in this paper. To enable this, the code, query sentences, and nearest neighbors per embedder and query will be publicly available. In this section, we present the from the experiment described in §4. Fig. 3 shows N2O between each pair of sentence embedders listed in Table 1; the values range from 0.04 to 0.62. While even the maximum observed value may not seem large, we reiterate that overlap is computed over two draws of k = 50 sentences (nearest neighbors) from approximately 8 million sentences, and even an N2O of 0.04 is unlikely from random chance alone. Averages of static word embeddings. We first observe that there is generally high N2O among this set of embedders in comparison to other categories (Fig. 4, left). Some cases where N2O is high for variations of the same embedder: glove-6b-100d and glove-6b-300d, which have different dimensionality but are otherwise trained with the same method and corpus (and to a lesser extent glove-840b-300d, which retains casing and is trained on a different corpus); fasttext-cc and fasttext-wiki, which again are trained with the same method, but different corpora. The use of subword information, unique to fasttext-cc-sub and fasttext-wiki-sub, has a large effect on N2O; there is a high (0.52) N2O value for these two and much lower N2O with other embedders, including their analogues without subword information. This effect is also illustrated by measuring, for a given embedder, the average token overlap between the query and its neighbors (see Fig. 5 in Appendix B). As we would expect, subword methods find near neighbors with lower token overlap, because they embed surface-similar strings near to each other. tf-idf. Unsurprisingly, tf-idf has low N2O with other embedders (even those based on static word embeddings). Like the subword case, we can also use token overlap to understand why this is the case: its nearest neighbors have by far the largest token overlap with the query (0.43). Averages of ELMo embeddings. We test three ELMo pretrained models across different capacities (elmo-small, elmo-orig) but the same training data, and across different training data but the same model capacity (elmo-orig, elmo-orig-5.5b). These two embedder pairs have high N2O (0.42 and 0.55 respectively); the mismatched pair, with both different training data and capacities, has slightly lower N2O (0.38). Figure 3: Heatmap of N2O for every pair of sentence embedders in Table 1 for k = 50, averaged across five samples of n = 100 queries; darker colors indicate higher overlap. A larger version of this plot (annotated with N2O values) is in Appendix B. BERT and GPT. We first find that specific-token representations for BERT or GPT (bert-base-cls, gpt-last) are outliers compared to other embedders (i.e., low N2O; see Fig. 4). This itself is not unexpected, as the training objectives for both of the pretrained models (without finetuning) are not geared towards semantic similarity the way other embedders are. What is surprising is that this effect seems to hold even for the MultiNLI-finetuned version of BERT (bert-ft-cls); if anything, this decreases N2O with other embedders further. 7 Notably, taking averaged BERT and GPT embeddings yields higher N2O with other embedders, especially ELMobased ones. Fig. 6 (Appendix B) plots the N2O values for each embedder compared to all others. Encoder-based embedders. We find that InferSent has highest N2O (∼0.2-0.3) with the embeddings based on averaging, despite InferSent being trained using a NLI task; that said, this is not wholly surprising as the model was initialized using GloVe vectors (glove-840b-300d) during training. The USE variants (DAN and Transformer) have fairly distinct nearest neighbors compared to other methods, with highest N2O between each other (0.24). Varying k. One possible concern is how sensitive our procedure is to k (the number of nearest neighbors from which overlap is computed): we would not want conflicting judgments of how similar two sentence embedders are due to different k. To confirm this, we first compute the ranked lists of N2O output for each k ∈ {5, 10, . . ., 45, 50}, where each list consists of all embedder pairs ordered by N2O for that k. We then compute Spearman's rank correlation coefficient (ρ) between each pair of ranked lists, where 1 indicates perfect positive correlation. We find that the average 7 In preliminary experiments, we also saw similar with BERT finetuned on the Microsoft Research Paraphrase Corpus ; that is, the effect does not seem specific to MultiNLI. Spearman's ρ is very high (0.996; min. 0.986) -i.e., the rankings of embedder similarity by N2O are reasonably stable across different values of k, even as far as k = 5 and k = 50. Query sampling. We also examine how the may vary across different query samples; as noted previously, the presented are averaged across five samples of n = 100 queries each. Standard deviations for N2O values across the five samples range from 0.005 to 0.019 (avg. 0.011). That is, given the range of N2O values being compared, the differences due to different query samples is small. We compute Spearman's ρ across different N2O samples in the same manner as above (k = 50) and find an average ρ of 0.994 (min. 0.991). Runtime. A theoretical concern with N2O is that, naively, its computation is linear in the size of the corpus, and to have reasonable semantic overlap within a diverse set of sentences, the corpus should be large. While our implementation of exact nearest neighbor search is sufficiently fast in practice, 8 we provide comments on use of approximate nearest neighbor methods in Appendix C. In the previous section, we performed a basic comparison between sentence embedders using N2O. Here, we show one kind of analysis enabled by N2O: given a query, which sentences from the corpus C are consistently its neighbors across different embedders? We might expect, for example, that a nearly identical paraphrase of the query will be a "popular" neighbor chosen by most embedders. Table 2 shows an example query with a sentence that is in the 5-nearest neighborhood for all sentence embedders, as well as sentences that are highly ranked for some embedder but not in the nearest neighbor sets for any other embedder (for larger k = 50). Qualitatively, what we find with this example's outlier sentences is that they are often thematically similar in some way (such as fiscal matters in Table 2), but with different participants. We also observe that extremely "popular" neighbors tend to have high lexical overlap with the query. Attempts to derive sentence embeddings that capture semantic similarity are inspired by the phenomenon of paraphrase; in this section, we use nearest neighbors to probe how sentence embedders capture paraphrase. More specifically, we carry out a "needle-in-a-haystack" experiment using the Semantic Textual Similarity Benchmark (STS;). STS contains sentence pairs with human judgments of semantic similarity on a 1-5 continuous scale (least to most similar). Query: Britain's biggest mortgage lender says that average house prices fell 3.6 percent in September, but analysts believe the market isn't that weak. Embedder Rank Sentence all embedders ≤ 5 Average house prices in Britain fell 3.6 percent in September from a month earlier, the country's biggest mortgage lender said Thursday, although analysts believe the market isn't that weak. bert-base-cls 6 Some analysts say that the December data indicate that consumer spending remains weak, making it harder for the economy to keep a sustained rebound. bert-ft-avg 5 An industry group says German machinery orders were down 3 percent on the year in January but foreign demand is improving. gpt-last 8 The economy has since rebound and grew 8.9 percent year-on-year in the second quarter, the central bank said last month, with growth expected to exceed six percent in the full year. Table 2: Popular and outlier near neighbors for the given query (top). The first sentence is in the 5-nearest neighborhood for all embedders; the remaining sentences are highly-ranked by the given embedder and outside the 50-nearest neighborhood for all other embedders. See Table 3 (Appendix B) for additional examples. We take 75 pairs in the 4-5 range from the STS development and test sets where the sentence pair has word-level overlap ratio < 0.6 -i.e., near paraphrases with moderately different surface semantics. We also constrain the sentence pairs to come from the newstext-based parts of the dataset. The first sentence in each sentence pair is the "query," and the second sentence is (temporarily) added to our Gigaword corpus. An example sentence pair, scored as 4.6, is: (A) Arkansas Supreme Court strikes down execution law and (B) Arkansas justices strike down death penalty. We then compute the rank of the sentence added to the corpus (i.e., the value of k such that the added sentence is part of the query's nearest neighbors). An embedder that "perfectly" correlates semantic similarity and distance should yield a rank of 1 for the sentence added to the corpus, since that sentence would be nearest to the query. Results. Using mean reciprocal rank (MRR), we find that the larger ELMo models and Infersent do particularly well at placing paraphrase pairs near each other. We also find that averaged BERT and GPT embeddings consistently perform better than the [CLS]/final token ones 9; this is consistent with our earlier observation (§5) that their training objectives may not yield specific-token embeddings that directly encode semantic similarity, hence why they are outliers by N2O. The full table of is in Table 4 (Appendix B). Recent comparisons of sentence embedders have been primarily either linguistic probing tasks or downstream evaluations. Linguistic probing tasks test whether embeddings can distinguish surface level properties, like sentence length; syntactic properties, like tree depth; and semantic properties, like coordination inversion. , ,, and , among others. Downstream evaluations are often classification tasks for which good sentence representations are helpful (e.g., NLI). Evaluations like the RepEval 2017 shared task , SentEval toolkit , and GLUE benchmark seek to standardize comparisons across sentence embedding methods. N2O is complementary to these, providing a task-agnostic way to compare embedders' functionality. In this paper, we introduce nearest neighbor overlap (N2O), a comparative approach to quantifying similarity between sentence embedders. Using N2O, we draw comparisons across 21 embedders. We also provide additional analyses made possible with N2O, from which we find high variation in embedders' treatment of semantic similarity. GloVe. We use three sets of standard pretrained GloVe embeddings: 100D and 300D embeddings trained on Wikipedia and Gigaword (6B tokens), and 300D embeddings trained on Common Crawl (840B tokens). 13 We handle tokenization and embedding lookup identically to word2vec; for the Wikipedia/Gigaword embeddings, which are uncased, we lower case all tokens as well. FastText. We use four sets of pretrained FastText embeddings: two trained on Wikipedia and other news corpora, and two trained on Common Crawl (each with an original version and one trained on subword information). 14 We use the Python port of the FastText implementation to handle tokenization, embedding lookup, and OOV embedding computation. ELMo. We use three pretrained models made available by AllenNLP: small, original, and original (5.5B). 16 We use spacy to perform word tokenization, consistent with the allennlp library; we also use allennlp (0.7.2) to compute the ELMo embeddings. We average the embeddings over all three bidirectional LSTM layers. BERT. We use Hugging Face's pytorch-transformers (0.6.2) implementation and pretrained BERT base cased model. 17 To tokenize, we use the provided BertTokenizer, which handles WordPiece (subword) tokenization, and in general follow the library's recommendations for feature extraction. For finetuning BERT on MultiNLI (matched subset), we generally use the default parameters provided in the library's run classifier.py (batch size = 32, learning rate = 5e-5, etc.). We finetune for three epochs, and obtain 84.1% dev accuracy (reasonably consistent with the original work). GPT. We use the same Hugging Face library and associated pretrained model for GPT; we use their BPE tokenizer and spacy for subword and word tokenization respectively. InferSent. We use the authors' implementation of InferSent, as well as their pretrained V1 model based on GloVe. 18 (Unfortunately, the FastText-based V2 model was not available while performing the experiments in this paper; see issues #108 and #124 in the linked Github.) As per their README, we use the nltk tokenizer (3.2.5). Universal Sentence Encoder. We use pretrained models available on TensorFlow Hub for both the DAN and Transformer variants. 19 The modules handle text preprocessing on their own. Experiments for ELMo, BERT, GPT, and the Transformer version of USE were run on a NVIDIA Titan XP GPU with CUDA 9.2. All other experiments were performed on CPUs. In §5, we noted that the FastText subword variants had much lower N2O compared to other embedders (including analogues without subword information). Fig. 5 shows average token overlap between a query and its nearest neighbors, averaged over all queries. Unsurprisingly, tfidf has by far the highest overlap, and fasttext-wiki-sub and fasttext-cc-sub the lowest. In §5, we found that the BERT and GPT based embedders had low N2O with all other embedders, and averaging (rather than taking the [CLS] or last embedding) generally raised N2O. Fig. 6 shows boxplots of N2O values between each embedder and all other embedders. Table 3 shows additional outlier nearest neighbors from Table 2. Query: Britain's biggest mortgage lender says that average house prices fell 3.6 percent in September, but analysts believe the market isn't that weak. Embedder Rank Sentence all embedders ≤ 5 Average house prices in Britain fell 3.6 percent in September from a month earlier, the country's biggest mortgage lender said Thursday, although analysts believe the market isn't that weak. bert-ft-cls 2 Japanese consumer prices fell for 13th straight month in March, though the GDP data suggests that deflationary pressures are starting to ease. fasttext-cc-sub 6 It cautioned however that the economic situation abroad could still slow Sweden's recovery, and said the country's gross domestic product (GDP) would grow just 3.6 percent in 2011, down from its May estimate of 3.7 percent growth. glove-840b-300d 12 Meanwhile, Australia's central bank left its key interest rate unchanged at 3.75 percent on Tuesday, surprising investors and analysts who had predicted the bank would continue raising the rate as the nation's economy rebounds. Table 3: Additional outlier near neighbors for the given query (top; same as Table 2). The first sentence is in the 5-nearest neighborhood for all embedders; the remaining sentences are highlyranked by the given embedder and outside the 50-nearest neighborhood for all other embedders. Table 4 shows for the query-paraphrase experiment in §8: mean reciprocal rank (MRR), the number of queries for which its paraphrase was its nearest neighbor, and the number of queries for which the paraphrase was in its 5-nearest neighborhood. Table 4: Results for the query-paraphrase experiment (§8), sorted by decreasing MRR. # top and # top-5 are the number of queries for which the paraphrase was the nearest neighbor and in the 5-nearest neighborhood (max. 75), respectively.

Represent the TLDR for finding the corresponding scientific article

We propose nearest neighbor overlap, a procedure which quantifies similarity between embedders in a task-agnostic manner, and use it to compare 21 sentence embedders.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: represent the article for finding a single-sentence tldr.

Question: Generative Adversarial Networks (GANs) can achieve state-of-the-art sample quality in generative modelling tasks but suffer from the mode collapse problem. Variational Autoencoders (VAE) on the other hand explicitly maximize a reconstruction-based data log-likelihood forcing it to cover all modes, but suffer from poorer sample quality. Recent works have proposed hybrid VAE-GAN frameworks which integrate a GAN-based synthetic likelihood to the VAE objective to address both the mode collapse and sample quality issues, with limited success. This is because the VAE objective forces a trade-off between the data log-likelihood and divergence to the latent prior. The synthetic likelihood ratio term also shows instability during training. We propose a novel objective with a ``"Best-of-Many-Samples" reconstruction cost and a stable direct estimate of the synthetic likelihood. This enables our hybrid VAE-GAN framework to achieve high data log-likelihood and low divergence to the latent prior at the same time and shows significant improvement over both hybrid VAE-GANS and plain GANs in mode coverage and quality. Generative Adversarial Networks (GANs) have achieved state-of-the-art sample quality in generative modeling tasks. However, GANs do not explicitly estimate the data likelihood. Instead, it aims to "fool" an adversary, so that the adversary is unable to distinguish between samples from the true distribution and the generated samples. This leads to the generation of high quality samples . However, there is no incentive to cover the whole data distribution. Entire modes of the true data distribution can be missedcommonly referred to as the mode collapse problem. In contrast, the Variational Auto-Encoders (VAEs) explicitly maximize data likelihood and can be forced to cover all modes . VAEs enable sampling by constraining the latent space to a unit Gaussian and sampling through the latent space. However, VAEs maximize a data likelihood estimate based on the L 1 /L 2 reconstruction cost which leads to lower overall sample quality -blurriness in case of image distributions. Therefore, there has been a spur of recent work (; ;) which aims integrate GANs in a VAE framework to improve VAE generation quality while covering all the modes. Notably in , GANs are integrated in a VAE framework by augmenting the L 1 /L 2 data likelihood term in the VAE objective with a GAN discriminator based synthetic likelihood ratio term. reports that in case of hybrid VAE-GANs, the latent space does not usually match the Gaussian prior. This is because, the reconstruction log-likelihood in the VAE objective is at odds with the divergence to the latent prior (also in case of alternatives proposed by ;). This problem is further exacerbated with the addition of the synthetic likelihood term in the hybrid VAE-GAN objective -it is necessary for sample quality but it introduces additional constraints on the encoder/decoder. This leads to the degradation in the quality and diversity of samples. Moreover, the synthetic likelihood ratio term is unstable during training -as it is the ratio of outputs of a classifier, any instability in the output of the classifier is magnified. We directly estimate the ratio using a network with a controlled Lipschitz constant, which leads to significantly improved stability. Our contributions in detail are, 1. We propose a novel objective for training hybrid VAE-GAN frameworks, which relaxes the constraints on the encoder by giving the encoder multiple chances to draw samples with high likelihood enabling it to generate realistic images while covering all modes of the data distribution, 2. Our novel objective directly estimates the synthetic likelihood term with a controlled Lipschitz constant for stability, 3. Finally, we demonstrate significant improvement over prior hybrid VAE-GANs and plain GANs on highly muti-modal synthetic data, CIFAR-10 and CelebA. Generative Autoencoders. VAEs allow for generation by maintaining a Gaussian latent space. , the Gaussian constraint in applied point-wise and latent representation of each point is forced towards zero. Adversarial Auto-encoders (AAE) and Wasserstein Auto-encoders (WAE) tackle this problem by an approximate estimate of the divergence which only requires the latent space to be Gaussian as a whole. But, the Gaussian constraint in;; ) is still at odds with the data log-likelihood. In this work, we enable the encoder to maintain both the latent representation constraint and high data log-likelihood using a novel objective. Furthermore, we integrate a GAN-based synthetic likelihood term to the objective to enhance the sharpness of generated images. Mode Collapse in Classical GANs. The classic GAN formulation has several shortcomings -importantly mode collapse. Denoising Feature Matching deals with the mode collapse by regularizing the discriminator using an auto-encoder. MDGAN uses two separate discriminators and regularizes using a auto-encoder. In EBGAN (a), the discriminator is interpreted as an energy functional and is also cast in an auto-encoder framework, leading to improvements in semi-supervised learning tasks. BEGAN proposes a Wasserstein distance based objective to train such GANs with auto-encoder based discriminators. The proposed approach leads to smoother convergence. InfoGAN maximizes the mutual information between a small subset of latent variables and observations in a Information Theoretic framework. This leads to disentangled and more interpretable latent representations. PacGAN proposes to deal with the mode collapse problem by using the discriminator to distinguish between product distributions. D2GAN proposes to use two discriminators -one for the forward KL divergence between the true and generated distributions and one for the reverse. BourGAN proposes to learn the distribution of the latent space (instead of assuming Gaussian) which reflects the distribution of the data. In , a inverse mapping from from latent to data space is learned and the generator is penalized based on the inverted distribution to cover all modes. proposes a moment matching paradigm different from VAEs or GANs. However, as the presented moment matching network involves an order of magnitude more parameters compared to VAEs or GANs, we do not consider them here. As we propose a hybrid VAE-GAN framework these techniques can be applied on top to potentially improve . However, in hybrid VAE-GANs the reconstruction loss already incentivizes the coverage of all modes. Wasserstein Loss based Formulations.; proposes GANs which minimize the Wasserstein distance between true and generated distributions. demonstrates improved by applying Spectral Normalization on the weights. , distance constraints are applied on top. WGANs were extended to Banach Spaces to emphasize edges or large scale behavior. focus on progressively learning to use more complex model architectures to improve performance. We use the regularization techniques developed for WGANs to improve stability of our hybrid VAE-GAN framework. shows very high quality generations at high resolutions but these are class conditional. However, diverse class conditional generation is considerably easier as intra-class variability is generally much lower than inter-class variability. Here, we focus on the more complex unconditional image generation task. Hybrid VAE-GANs. a VAE-GAN hybrid is proposed with discriminator feature matching -the VAE decoder is trained to match discriminator features instead of a L 1 /L 2 reconstruction loss. ALI proposes to instead match the encoder and decoder joint distributions -with limited success on diverse datasets. BiGAN , builds upon ALI to learn inverse mappings from the data to the latent space and demonstrate effectiveness on various discriminative tasks. extends standard VAEs by replacing the loglikelihood term with a hybrid version based on synthetic likelihoods. The KL-divergence constraint to the prior is also recast to a synthetic likelihood form, which can be enforced by a discriminator (as in ;). The second improvement is crucial in generating realistic images at par with classic/Wasserstein GANs. We further improve upon by allowing the encoder multiple chances to draw desired samples and enforcing stability -enabling it to maintain low divergence to the prior while generating realistic images. We begin with a brief overview of hybrid VAE-GANs followed by details of our novel objective. Overview. Hybrid VAE-GANs (Figure 1) are generative models for data distributions x ∼ p(x) that transform a latent distribution z ∼ p(z) to a learned distributionx ∼ p θ (x) approximating p(x). The GAN (G θ,D I alone can generate realistic samples, but has trouble covering all modes. The VAE (R φ,G θ,D L) can cover all modes of the distribution, but generates lower quality samples overall. VAE-GANs leverage the strengths of both VAEs and GANs to generate high quality samples while capturing all modes. We begin with a discussion of the prior hybrid VAE-GAN objectives and its shortcomings, followed by our novel "Best-of-Many-Samples" objective with a novel reconstruction term and regularized stable direct estimate of the synthetic likelihood. Hybrid VAE-GANs (; ; ; b) maximizes the log-likelihood of the data (x ∼ p(x)) akin to VAEs. The log-likelihood, assuming the latent space to be distributed according to p(z), Here, p(z) is usually Gaussian. This requires the generator G θ to generate samples that assign high likelihood to every example x in the data distribution for a likely z ∼ p(z). Thus, the decoder θ can be forced to cover all modes of the data distribution x ∼ p(x). In contrast, GANs never directly maximize the data likelihood and there is no direct incentive to cover all modes. However, the integral in is intractable. VAEs and Hybrid VAE-GANs use amortized variational inference using a recognition network q φ (z|x) (R φ). The final hybrid VAE-GAN objective of the state-of-the-art α-GAN which integrates a synthetic likelihood ratio term is, This objective has two important shortcomings. Firstly, as pointed in , this objective severely constrains the recognition network as the average likelihood of the samples generated from the posterior q φ (z|x) is maximized. This forces all samples from q φ (z|x) to explain x equally well, penalizing any variance in q φ (z|x) and thus forcing it away from the Gaussian prior p(z). Therefore, this makes it difficult to match the prior in the latent space and the encoder is forced to trade-off between a good estimate of the data log-likelihood and the divergence to the latent prior. Secondly, the synthetic likelihood ratio term is the ratio of the output of D I, any instability (nonsmoothness) in the output of the classifier is magnified. Moreover, there is no incentive for D I to be smooth (stable). For two similar images, {x 1, x 2} with |x 1 − x 2 | ≤, the change of output |D I (x 1 |z 1) − D I (x 1 |z 2)| can be arbitrarily large. This means that a small change in the generator output (e.g. after a gradient descent step) can have a large change in the discriminator output. Next, we describe how we can effectively leverage multiple samples from q φ (z|x) to deal with the first issue. Finally, we derive a stable synthetic likelihood term to deal with the second issue. Figure 1: Overview of our BMS-VAE-GAN framework. The terms of our novel objective are highlighted at the right. We consider only the best sample from the generator G θ while computing the reconstruction loss. Building upon , we derive an alternative variational approximation of, which uses multiple samples to relax the constrains on the recognition network (full derivation in Appendix A), In comparison to the α-GAN objective where the expected likelihood assigned by each sample to the data point x was considered, we see that in the likelihood is computed considering all generated samples. The recognition network gets multiple chances to draw samples which assign high likelihood to x. This allows q φ (z|x) to have higher variance, helping it better match the prior and significantly reducing the trade-off with the data log-likelihood. Next, we describe how we can integrate a synthetic likelihood term in to help us generate sharper images. Considering only L 1 /L 2 reconstruction based likelihoods p θ (x|z) (as in ; ;) might not be sufficient in case of complex high dimensional distributions e.g. in case of image data this leads to blurry samples. Synthetic estimates of the likelihood leverages a neural network (usually a classifer) which is jointly trained to distinguish between real and generated samples. The network is traiend to assign low likelihood to generated samples and higher likelihood to real data samples. Starting from, we integrate a synthetic likelihood term with weight β to encourage our generator to generate realistic samples. The L 1 /L 2 reconstruction likelihood (with weight α) forces the coverage of all modes. However, unlike prior work , our synthetic likelihood estimator D I is not a classifier. We first convert the likelihood term to a likelihood ratio form which allows for synthetic estimates, To enable the estimation of the likelihood ratio p θ (x|z) /p(x) using a neural network, we introduce the auxiliary variable y where, y = 1 denotes that the sample was generated and y = 0 denotes that the sample is from the true distribution. We can now express (using Bayes theorem, see Appendix A), The ratio p θ (y=1|z,x) /1−p(y=1|x) should be high for generated samples which are indistinguishable from real samples and low otherwise. In case of image distributions, we find that direct estimation of the numerator/denominator (as in) exacerbates instabilities (non-smoothness) of the estimate. Therefore, we estimate this ratio directly using the neural network D I (x) -trained to produce high values for images indistinguishable from real images and low otherwise, To further unsure smoothness, we directly control the Lipschitz constant K of D I. This ensures, | -the function is strictly smooth everywhere. Small changes in generator output cannot arbitrarily change the synthetic likelihood estimate, hence allowing the generator to smoothly improve sample quality. We constrain the Lipschitz constant K to 1 using. Note that the likelihood p θ (x|z) takes the form e −λ x−x n in -a log-sum-exp which is numerically unstable. As we perform stochastic gradient descent, we can deal with this after stochastic (MC) sampling of the data points. We can well estimate the log-sum-exp using the max -the "Best-of-Many-Samples" , In practice, we observe that we can improve sharpness of generated images by penalizing generator G θ, using the least realistic of the T samples, Our final "Best-of-Many"-VAE-GAN objective takes the form (ignoring the constant log(T) term), We use the same optimization scheme as in. We provide the algorithm in detail in Appendix B. Approximation Errors. The "Best-of-Many-Samples" scheme introduces the log(T) error term. However, this error term is dominated by the low data likelihood term in the beginning of optimization . Later, as generated samples become more diverse, the log likelihood term is dominated by the Best of T samples -"Best of Many-Samples" is equivalent. Classifier based estimate of the prior term. Recent work has shown that point-wise minimization of the KL-divergence using its analytical form leads to degradation in image quality. Instead, KL-divergence term is recast in a synthetic likelihood ratio form minimized "globally" using a classifier instead of point-wise. Therefore, unlike , here we employ a classifier based estimate of the KL-divergence to the prior. However, as pointed out by prior work on hybrid VAE-GANs , a classifier based estimate with still leads to mismatch to the prior as the trade-off with the data log-likelihood still persists without the use of the "Best-of-Many-Samples". Therefore, as we shall demonstrate next, the benefits of using the "Best-of-Many-Samples" extends to case when a classifier based estimate of the KL-divergence is employed. Next, we evaluate on multi-modal synthetic data as well as CIFAR-10 and CelebA. We perform all experiments on a single Nvidia V100 GPU with 16GB memory. We use as many samples during training as would fit in GPU memory so that we make the same number of forward/backward passes as other approaches and minimize the computational overhead of sampling multiple samples. Table 2: Visualization of samples. The standard WAE and α-GAN objectives leads to mismatch to the prior in the latent space. We show samples z (in red) which are highly likely under the standard Gaussian prior (blue) but have low probability under the learnt marginal posterior q φ (z). Bottom Row: We show that such points z lead to low quality data samples (in red), which do correspond to any of the modes. We evaluate in Tables 1 and 2 on the standard 2D Grid and Ring datasets, which are highly challenging due to their multi-modality. The metrics considered are the number of modes captured and % of high quality samples (within 3 standard deviations of a mode). The generator/discriminator architecture is same as in. We see that our BMS-VAE-GAN (using the best of T = 10 samples) outperforms state of the art GANs e.g. and the WAE and α-GAN baselines. The explicit maximization of the data log-likelihood enables our BMS-VAE-GAN and the WAE and α-GAN baselines to capture all modes in both the grid and ring datasets. The significantly increased proportion of high quality samples with respect to WAE and α-GAN baselines is due to our novel "Best-of-Many-Samples" objective. We illustrate this in Table 3. we analyze the learnt latent spaces in detail, in particular we check for points (in red) which are likely under the Gaussian prior p(z) (blue) but have low probability under the marginal posterior q φ (z) = q φ (z|x)dx. We use tSNE to project points from our 32-dimensional latent space to 2D. In Table 3 (Top Row) we clearly see that there are many such points in case of the WAE and α-GAN baselines (note that this low probability threshold is common across all methods). In Table 3 (Bottom Row) we see that these points lead to the generation of low quality samples (in red) in the data space. Therefore, we see that our "Best-of-Many-Samples" samples objective helps us match the prior in the latent space and thus this leads to the generation of high quality samples and outperforming both state of the art GANs and hybrid VAE-GAN baselines. 0.0055±0.0006 α-GAN + SN (Ours) T = 1 0.0048±0.0005 BMS-VAE-GAN (Ours) T = 30 0.0037±0.0005 Table 5: Closest generated images found using IvOM. Next, we evaluate on the CIFAR-10 dataset. Auto-encoding based approaches do not perform well on this dataset, as a simple Gaussian reconstruction based likelihood is insufficient for such highly multi-modal image data. This makes CIFAR-10 a very challenging dataset for hybrid VAE-GANs. Architecture Details. We use two different types of architectures for the generator/discriminator pair G θ, D I: DCGAN based as used in and the Standard CNN used in;. Experimental Details and Baselines. We use the ADAM optimizer and use learning rate of 2 × 10 −4, β 1 = 0.0 and β 2 = 0.9 for all components. We use the same architecture of the latent space discriminator D L as in α- We consider the following baselines for comparison against our BMS-VAE-GAN with a DCGAN generator/discriminator, 1. A standard DCGAN , 2. The α-GAN model of . Furthermore, we compare our BMS-GAN with the Standard CNN generator/discriminator to, 1. SN-GAN , 2. BW-GAN , 3. Dist-GAN , 4. Our α-GAN + SN is an improved version of the α-GAN which includes Spectral Normalization for stable estimation of synthetic likelihoods. Again, the α-GAN and α-GAN + SN baselines are identical to the corresponding BMS-VAE-GAN except for the "Best-of-Many-Samples" reconstruction likelihood. Method FID ↓ DCGAN Architecture WAE 89.3±0.3 BMS-VAE (Ours) T = 10 87.9±0.4 DCGAN 30.7±0.2 α-GAN 29.4±0.3 BMS-GAN (ours) T = 10 28.8±0.4 Standard CNN Architecture SN-GAN 25.5 BW-GAN 25.1 α-GAN + SN (Ours) T = 1 24.6±0.3 BMS-VAE-GAN (Ours) T = 10 23.8±0.2 BMS-VAE-GAN (Ours) T = 30 23.4±0.2 Dist-GAN 22.9 BMS-VAE-GAN (Ours) T = 10 21.8±0.2 Table 6: FID on CIFAR-10. Table 6. Please note that the higher latent space dimensionality makes the latent spaces much harder to reliably analyze. Therefore, we rely on the FID and IoVM metrics. We follow evaluation protocol of; and use 10k/5k real/generated samples to compute the FID score. The α-GAN model with (DCGAN architecture) demonstrates better fit to the true data distribution (29.3 vs 30.7 FID) compared to a plain DCGAN. This again shows the ability of hybrid VAE-GANs in improving the performance of plain GANs. We observe that our novel "Best-of-ManySamples" optimization scheme outperforms both the plain DCGAN and hybrid α-GAN(28.8 vs 29.4 FID), confirming the advantage of using "Best-of-Many-Samples". Furthermore, we see that our BMS-VAE outperforms the state-of-the-art plain auto-encoding WAE . We further compare our BMS-VAE-GAN to state-of-the-art GANs using the Standard CNN architecture in Table 6 with 100k generator iterations. Our α-GAN + SN ablation significantly outperforms the state-of-the-art plain GANs , showing the effectiveness of hybrid VAE-GANs with a stable direct estimate of the synthetic likelihood on this highly diverse dataset. Furthermore, our BMS-VAE-GAN model trained using the best of T = 30 samples significantly improves over the α-GAN + SN baseline (23.4 vs 24.6 FID), showing the effectiveness of our "Best-of-Many-Samples". We also compare to using 300k generator iterations, again outperforming by a significant margin (21.8 vs 22.9 FID). The IoVM metric of (Tables 4 and 5), illustrates that we are also able to better reconstruct the image distribution. The improvement in both sample quality as measured by the FID metric and data reconstruction as measured by the IoVM metric shows that our novel "Best-of-Many-Samples" objective helps us both match the prior in the latent space and achieve high data log-likelihood at the same time. Next, we evaluate on CelebA at resolutions 64×64 and 128×128. Training and Architecture Details. As the focus is to evaluate objectives for hyrid VAE-GANs, we use simple DCGAN based generators and discriminators for generation at both 64×64 and 128×128. Approaches like progressive growing are orthogonal and can be applied on top. Baselines and Experimental Details. We consider the following baselines for comparison with our BMS-GAN with T = {10, 30} samples, 1. WAE the state-of-the-art plain auto-encoding generative model, 2. α-GAN the state-of-the-art hybrid VAE-GAN, 3. Our α-GAN + SN is an improved version of the α-GAN which includes Spectral Normalization for stable estimation of synthetic likelihoods. Note, the α-GAN baseline is identical to our BMS-GAN except for the "Best-of-Many" reconstruction likelihood. Moreover, we include several plain GAN baselines, 1. Wasserstein GAN with gradient penalty (WGAN-GP) , 2. Spectral Normalization GAN (SN-GAN) , 3. Dist-GAN . To train our BMS-VAE-GAN and αR-GAN models we use the two time-scale update rule with learning rate of 1 × 10 −4 for the generator and 4 × 10 −4 for the discriminator. We use the Adam optimizer with β 1 = 0.0 and β 2 = 0.9. We use a three layer MLP with 750 neurons as the latent space discriminator D L (as in) and a DCGAN based recognition network R φ. We use the hinge loss to train D I to produce high values for real images and low values for generated images, log(D I) ∈ [−0.5, 0.5] works well. 26.3±0.9 Dist-GAN 23.7±0.3 SN-GAN 21.9±0.8 α-GAN 19.2±0.8 α-GAN + SN (Ours) T = 1 15.1±0.2 BMS-VAE-GAN (Ours) T = 10 14.3±0.4 BMS-VAE-GAN (Ours) T = 30 13.6±0.4 Resolution: 128×128 SN-GAN 60.5±1.5 αR-GAN (Ours) T = 1 45.8±1.4 BMS-GAN (Ours) T = 10 42.7±1.2 Table 7: FID on CelebA. We train all models for 200k iterations and report the FID scores for all models using 10k/10k real/generated samples in Table 7. The pure auto-encoding based WAE has the weakest performance due to blurriness. Our pure autoencoding BMS-VAE (without synthetic likelihoods) improves upon the WAE (39.8 vs 41.2 FID), already demonstrating the effectiveness of using "Best-of-Many-Samples". We see that the base DCGAN has the weakest performance among the GANs. BEGAN suffers from partial mode collapse. The SN-GAN improves upon WGAN-GP, showing the effectiveness of Spectral Normalization. However, there exists considerable artifacts in its generations. The α-GAN of , which integrates the base DCGAN in its framework performs significantly better (31.1 vs 19.2 FID). This shows the effectiveness of VAE-GAN frameworks in increasing quality and diversity of generations. Our enhanced α-GAN + SN regularized with Spectral Normalization performs significantly better (15.1 vs 19.2 FID). This shows the effectiveness of a regularized direct estimate of the synthetic likelihood. Using the gradient penalty regularizer of lead to drop of 0.4 FID. Our BMS-VAE-GAN improves significantly over the α-GAN + SN baseline using the "Best-of-Many-Samples" (13.6 vs 15.1 FID). The at 128×128 resolution mirror the at 64×64. We additionally evaluate using the IoVM metric in Appendix C. We see that by using the "Best-of-Many-Samples" we obtain sharper (Figure 4d) that cover more of the data distribution as shown by both the FID and IoVM. We propose a new objective for training hybrid VAE-GAN frameworks which overcomes key limitations of current hybrid VAE-GANs. We integrate, 1. A "Best-of-Many-Samples" reconstruction likelihood which helps in covering all the modes of the data distribution while maintaining a latent space as close to Gaussian as possible, 2. A stable estimate of the synthetic likelihood ratio.. Our hybrid VAE-GAN framework outperforms state-of-the-art hybrid VAE-GANs and plain GANs in generative modelling on CelebA and CIFAR-10, demonstrating the effectiveness of our approach. We begin with a derivation of the multi-sample objective. We maximize the log-likelihood of the data (x ∼ p(x)). The log-likelihood, assuming the latent space to be distributed according to p(z), Here, p(z) is usually Gaussian. However, the integral in is intractable. VAEs and Hybrid VAEGANs use amortized variational inference using an (approximate) variational distribution q φ (z|x) (jointly learned), To arrive at a tractable objective, the standard VAE objective applies the Jensen inequality at this stage, but this forces the final objective to consider the average data-likelihood. , we apply the Mean Value theorem of Integration to leverage multiple samples, We can lower bound with the minimum value of z, As the term on the right is difficult to estimate, we approximate it using the KL divergence (as in). Intuitively, as the KL divergence heavily penalizes q φ (z|x) if it is high for low values p(z), this ensures that the ratio p(z) /q φ (z |x) is maximized. Similar to , this leads to the "many-sample" objective of the main paper, Next, we provide a detailed derivation of. Again, to enable the estimation of the likelihood ratio p θ (x|z) /p(x) using a neural network, we introduce the auxiliary variable y where, y = 1 denotes that the sample was generated and y = 0 denotes that the sample is from the true distribution. We can now express as (using Bayes theorem), This is because, (assuming independence p(z, and, Assuming, p(y = 0) = p(y = 1) (equally likely to be true or generated), Update D I using hinge loss to produce high values (≥ a) for real images and low (≤ b) otherwise: Update D L using the standard cross-entropy loss: We detail in algorithm 1, how the components R φ, G θ, D I, D L of our BMS-VAE-GAN (see Figure Figure 1) are trained. We follow in designing algorithm 1. However, unlike , we train R φ, G θ jointly as we found it to be computationally cheaper without any loss of performance. Also unlike , we use the hinge loss to update D I as it leads to improved stability (as discussed in the main paper). We additionally evaluate using the IoVM on CelebA in Table 8, using the base DCGAN architecture at 64×64 resolution. We observe that our BMS-VAE-GAN performs better. The improvement is smaller compared to CIFAR-10 because CelebA is less multi-modal compared to CIFAR-10. However, we still observe better overall sample quality from our BMS-VAE-GAN. This means that although difference in data reconstruction is smaller, our BMS-VAE-GAN enables better match the prior in the latent space. Finally, we provide additional examples of closest matches found using IoVM in Figure 3, illustrating regions of the data distribution captured by BMS-VAE-GAN but not captured by SN-GAN or α-GAN + SN. IoVM ↓ SN-GAN 0.0221±0.0003 α-GAN + SN (Ours) T = 1 0.0036±0.0001 BMS-VAE-GAN (Ours) T = 10 0.0034±0.0001 In Figure 4, we compare qualitatively our BMS-VAE-GAN against other state-of-the-art GANs. We use the same settings as in the main paper and use the same DCGAN architecture across methods (as the aim is to evaluate training objectives). Again note that, approaches like use more larger generator/discriminator architectures and can be applied on top. We see that BEGAN produces sharp images (with only a few very minuscule artifacts), but lack diversity -also reflected by the lower FID score in Table 2 of the main paper. In comparison, both SN-GAN and Dist-GAN produce sharp and diverse images (again reflected by the FID score in Table 2 of the main paper) but also introduce artifacts. Dist-GAN introduces relatively fewer artifacts in comparison to SN-GAN . In comparison, our BMS-VAE-GAN strikes the best balance -generating sharp and diverse images with few if any artifacts (also again reflected by the FID scores in the main paper). We also provide additional qualitative examples on CIFAR-10 in Figure 5, highlighting sharper images compared to α-GAN +SN. In Table 9 we include diversity using the LPIPS metric. To compute the LPIPS diversity score 5k samples were randomly generated and the similarity within the batch was computed. We see that our BMS-VAE-GAN generates the most diverse examples on both datasets, further highlighting the effectiveness of our "Best-of-Many-Samples" objective. Table 9: Evaluation using the LPIPS metric.

We propose a new objective for training hybrid VAE-GANs which lead to significant improvement in mode coverage and quality.

Represent this text to find a much much longer article about it with say hundreds-thousands of words!

We propose a novel, projection based way to incorporate the conditional information into the discriminator of GANs that respects the role of the conditional information in the underlining probabilistic model.


Question: Represent the scientific article to find a short TLDR

Question: In order to mimic the human ability of continual acquisition and transfer of knowledge across various tasks, a learning system needs the capability for life-long learning, effectively utilizing the previously acquired skills. As such, the key challenge is to transfer and generalize the knowledge learned from one task to other tasks, avoiding interference from previous knowledge and improving the overall performance. In this paper, within the continual learning paradigm, we introduce a method that effectively forgets the less useful data samples continuously across different tasks. The method uses statistical leverage score information to measure the importance of the data samples in every task and adopts frequent directions approach to enable a life-long learning property. This effectively maintains a constant training size across all tasks. We first provide some mathematical intuition for the method and then demonstrate its effectiveness with experiments on variants of MNIST and CIFAR100 datasets. It is a typical practice to design and optimize machine learning (ML) models to solve a single task. On the other hand, humans, instead of learning over isolated complex tasks, are capable of generalizing and transferring knowledge and skills learned from one task to another. This ability to remember, learn and transfer information across tasks is referred to as lifelong learning or continual learning BID16 BID3 BID11. The major challenge for creating ML models with lifelong learning ability is that they are prone to catastrophic forgetting BID9 BID10. ML models tend to forget the knowledge learned from previous tasks when re-trained on new observations corresponding to a different (but related) task. Specifically when a deep neural network (DNN) is fed with a sequence of tasks, the ability to solve the first task will decline significantly after training on the following tasks. The typical structure of DNNs by design does not possess the capability of preserving previously learned knowledge without interference between tasks or catastrophic forgetting. There have been different approaches proposed to address this issue and they can be broadly categorized in three types: I) Regularization: It constrains or regularizes the model parameters by adding some terms in the loss function that prevent the model from deviating significantly from the parameters important to earlier tasks. Typical algorithms include elastic weight consolidation (EWC) BID4 and continual learning through synaptic intelligence (SynInt) BID19. II) Architectural modification: It revises the model structure successively after each task in order to provide more memory and additional free parameters in the model for new task input. Recent examples in this direction are progressive neural networks BID14 and dynamically expanding networks BID18. III) Memory replay: It stores data samples from previous tasks in a separate memory buffer and retrains the new model based on both the new task input and the memory buffer. Popular algorithms here are gradient episodic memory (GEM) BID8, incremental classifier and representation learning (iCaRL) BID12.Among these approaches, regularization is particularly prone to saturation of learning when the number of tasks is large. The additional / regularization term in the loss function will soon lose its competency when important parameters from different tasks are overlapped too many times. Modifications on network architectures like progressive networks resolve the saturation issue, but do not scale as number and complexity of tasks increase. The scalability problem is also present when using memory replay and often suffer from high computational and memory costs. In this paper, we propose a novel approach to lifelong learning with DNNs that addresses both the learning saturation and high computational complexity issues. In this method, we progressively compresses the input information learned thus far along with the input from current task and form more efficiently condensed data samples. The compression technique is based on the statistical leverage scores measure, and it uses frequent directions idea in order to connect the series of compression steps for a sequence of tasks. Our approach resembles the use of memory replay since it preserves the original input data samples from earlier tasks for further training. However, our method does not require extra memory for training and is cost efficient compared to most memory replay methods. Furthermore, unlike the importance assigned to model specific parameters when using regularization methods like EWC or SynInt, we assign importance to the training data that is relevant in effectively learning new tasks, while forgetting less important information. Before presenting the idea, let's first setup the problem: DISPLAYFORM0..} represent a sequence of tasks, each task consists of n i data samples and each sample has a feature dimension d and an output dimension m, i.e., input A i ∈ R ni×d and true output B i ∈ R ni×m. Here, we assume the feature and output dimensions are fixed for all tasks 1. The goal is to train a DNN over the sequence of tasks and ensure it performs well on all of them. Here, we consider that the network's architecture stays the same and the tasks are received in a sequential manner. Formally, with f representing a DNN, our objective is to minimize the loss 2: DISPLAYFORM1 Under this setup, let's look at some existing models: Online EWC trains f on task (A i, B i) with a loss function containing additional penalty terms min f f (DISPLAYFORM2 j=1 Λ j and each Λ j is defined as the change of important parameters (using Fisher information matrix) in f with respect to the jth task. GEM keeps an extra memory buffer containing data samples from each of the previous tasks M k with k < i, it trains on the current task (A i, B i) with a regular loss func- DISPLAYFORM3, but subject to inequalities on each update of f, DISPLAYFORM4 The new approach OLSS is to find an approximation of A in a streaming manner, i.e., to form anÂ i to approximate DISPLAYFORM5 T such that the inĝ DISPLAYFORM6 is likely to perform on all tasks as good as DISPLAYFORM7 To avoid extra memory and computation cost during the training process, we restrict the approximateÂ i to have the same number of rows as the current task A i.Equation and represent nonlinear least squares problems. It is to be noted that a nonlinear least squares problem can be solved with an approximation deduced from an iteration of linear least squares problems with J T J∆θ = J T ∆B where J is the Jacobian of f at each update (Gauss-Newton Method). Besides this technique, there are various approaches in addressing this problem. Here we adopt a cost effective simple randomization technique -leverage score sampling, which has been used extensively in solving large scale linear least squares and low rank approximation problems BID17 BID0. Definition 1 BID1 ) Given a matrix A ∈ R n×d with n > d, let U denote the n × d matrix consisting of the d left singular vectors of A, and let U (i,:) denote the i-th row of U, then the statistical leverage score of the i-th row of A is defined as U (i,:) 2 2 for i ∈ {1, ..., n}.Statistical leverage scores define the relevant non-uniformity structure of a matrix and a higher score indicates a heavier weight of the row contributing to the non-uniformity of the matrix; it has been widely used for constructing a randomized sketch of a matrix BID1 BID17. In our case, given an input matrix A, we will compute the leverage score of each row, then sample the rows with probability proportional to the scores. Using leverage score sampling, we are able to select the important samples given a dataset. The remaining problem is to embed it in a sequence of tasks. In order to achieve this, we make use of the concept of frequent directions. Frequent directions extends the idea of frequent items in item frequency approximation problem to a matrix BID7 BID2 BID15. Given a matrix A ∈ R n×d whose rows are received one by one and a space parameter, the algorithm considers the first 2 rows in A and shrinks its top orthogonal vectors by the same amount to obtain an × d matrix; then combines them with the next rows in A for the next iteration, repeat the procedure until reaching the final sketch of dimension × d. Frequent directions algorithm is targeted at finding a low rank approximation on a continuously expanding matrix. This is well suited for a continuous stream of data (tasks) within the lifelong learning setting. We present the step by step procedure of performing leverage score sampling together with compression using frequent directions idea in Algorithm 1. In our setting, we append the new task data samples to the existing buffer set and perform leverage score sampling to form a new buffer set and then train on it, this process is repeated for the entire sequence of tasks. Input: A sequence of tasks {FIG1, ..., (A i, B i),...} with A i ∈ R ni×d and B i ∈ R ni×m; initialization of the model parameters; a space parameter i.e., number of samples to pass in the model for training. It can be set as n i or even smaller after receiving the i-th task, which avoids extra memory and computations during training. Output: A trained neural network on a sequence of tasks. Step 1 Initialize a buffer set S = {Â,B} where bothÂ andB are empty. Step 2 While the ith task is presented:Step 3 IfÂ andB are empty: Step 4 setÂ = A i andB = B i, Step 5 else:Step 6 setÂ = Â A i andB = B B i.Step 7 Perform SVD: DISPLAYFORM0 Step 8 Randomly select rows ofÂ andB without replacement based on probability U j,:2 2 / U 2 F for j ∈ {1, ..., n i +} (or j ∈ {1, ..., n i} when i = 1) and set them asÂ andB respectively. Step 9Train the model withÂ ∈ R ×d andB ∈ R ×m. When n i is large, the SVD (singular value decomposition) of matrixÂ ∈ R (ni+)×d in Step 6 is computationally expensive, we could use a streaming SVD method to speed up the process if is chosen much smaller than n i. In that case the computational cost for SVD could be reduced from O((n i +)d2 ) to O(log 2 (n i +) d 2 ) (assuming d < < n i). In addition, there exists various efficient ways to approximate the leverage scores BID1 BID13 which would further reduce the computational cost. Remark: A major concern with this algorithm is that leverage scores is a linear measure, i.e., the selected samples capture the important information embedded linearly in the data matrix which may not fully represent the importance of the data samples. Another related issue is that the nonlinear information probably depend on the structure of f, the DNN. As such, there may be some underlying dependency of a data sample's importance on the DNN architecture. We leave this open as a future research direction. We evaluate the performance of the proposed algorithm OLSS on three classification tasks used as benchmarks in related prior work.• Rotated MNIST : a variant of the MNIST dataset of handwriten digits BID6, the digits in each task are rotated by a fixed angle between 0 • to 180•. The experiment is on 20 tasks and each task consists of 60, 000 training and 10, 000 testing samples.• Permutated MNIST BID4: a variant of the MNIST dataset BID6, the digits in each task are transformed by a fixed permutation of pixels. The experiment is on 20 tasks and each task consists of 60, 000 training and 10, 000 testing samples.• Incremental CIFAR100 BID12 BID19: a variant of the CIFAR object recognition dataset with 100 classes BID5 ). The experiment is on 20 tasks and each task consists of 5 classes; each task consists of 2, 500 training and 500 testing samples. Where, each task introduces a new set of classes; for a total number of 20 tasks, each new task concerns examples from a disjoint subset of 5 classes. In the original setting of , a softmax layer is added to the output vector which only allows entries representing the 5 classes in the current task to output values larger than 0. In our setting, we allow the entries representing all the past occurring classes to output values larger than 0. We believe this is a more natural setup for lifelong learning. The DNN used for rotated and permuted MNIST is an MLP with 2 hidden layers and each with 400 units; whereas a ResNet18 is used for the incremental CIFAR100 experiment. We train 5 epochs with batch size 200 on rotated and permuted MNIST datasets and 10 epochs with batch size 100 on incremental CIFAR100. In all experiments we compare the following algorithms: I) A simple SGD predictor, II) EWC BID4, III) GEM and IV) OLSS (ours).In all the algorithms, we use a plain SGD optimizer. All algorithms were implemented based on the publicly available code from the original authors of the GEM paper BID8. The regularization and memory hyper-parameters in EWC and GEM were set as described in BID8. The space parameter for our OLSS algorithm was set to be equal to the number of samples in each task; the learning rate for each algorithm was determined through a grid search on {0.001, 0.003, 0.01, 0.03, 0.1, 0.3, 1.0}. Comparing across all the algorithms, we summarize the average test accuracy on the learned tasks in FIG1 (see Appendix Figure 2 for the change in the test accuracy at the first task, as more tasks are learned.) and the computational costs for each algorithm in TAB0. As observed from the figures, across the three benchmarks, OLSS and GEM achieve similar accuracy and significantly outperform both EWC and simple SGD training. Nevertheless, GEM demands much higher computational resources (see TAB0) as the algorithm requires a constraint validation step and a potential gradient projection step to correct for constraint violations across all previously learned tasks during training (see Section 3 in BID8). In detail, for GEM, the time complexity is proportional to the product of the number of samples kept in the memory buffer, the number of parameters in the model and the number of iterations required to converge. In contrast, OLSS requires a SVD (or QR factorization) to compute the leverage scores for each task which can be achieved in a time complexity proportional to the product of the square of the number of features and the number of data samples, and is much less compared to GEM. As observed in Appendix Figure 2, OLSS shows robustness to catastrophic forgetting of the first task with positive backward transfer across all three datasets while learning the remaining sequence of tasks. In the case of rotated and permuted MNIST, OLSS is the most robust method. As presented in Appendix Figure 3, after training on the whole sequence of tasks, both GEM and OLSS are able to preserve the accuracy for most tasks on rotated and permuted MNIST. In contrast, it is hard to preserve the accuracy of the previously trained tasks on CIFAR100 for all algorithms. As we noted earlier, EWC exhibits a saturation issue when the number of tasks increases. This may hold for most regularization methods in order to achieve continual learning, as they target constraining the model parameters successively, thereby limiting the model capacity. We presented a new approach in addressing the lifelong learning problem with deep neural networks. It is inspired by the randomization and compression techniques typically used in statistical analysis. We combined a simple importance sampling technique -leverage score sampling with the frequent directions concept and developed an online effective forgetting or compression mechanism that enables lifelong learning across a sequence of tasks. Despite its simple structure, the on MNIST and CIFAR100 experiments show its effectiveness as compared to recent state of the art.

A new method uses statistical leverage score information to measure the importance of the data samples in every task and adopts frequent directions approach to enable a life-long learning property.

we define the filter-level pruning problem for binary neural networks for the first time and propose method to solve it.