Question: Represent the scientific article to find a short TLDR

Question: Learned representations of source code enable various software developer tools, e.g., to detect bugs or to predict program properties. At the core of code representations often are word embeddings of identifier names in source code, because identifiers account for the majority of source code vocabulary and convey important semantic information. Unfortunately, there currently is no generally accepted way of evaluating the quality of word embeddings of identifiers, and current evaluations are biased toward specific downstream tasks. This paper presents IdBench, the first benchmark for evaluating to what extent word embeddings of identifiers represent semantic relatedness and similarity. The benchmark is based on thousands of ratings gathered by surveying 500 software developers. We use IdBench to evaluate state-of-the-art embedding techniques proposed for natural language, an embedding technique specifically designed for source code, and lexical string distance functions, as these are often used in current developer tools. Our show that the effectiveness of embeddings varies significantly across different embedding techniques and that the best available embeddings successfully represent semantic relatedness. On the downside, no existing embedding provides a satisfactory representation of semantic similarities, e.g., because embeddings consider identifiers with opposing meanings as similar, which may lead to fatal mistakes in downstream developer tools. IdBench provides a gold standard to guide the development of novel embeddings that address the current limitations. Reasoning about source code based on learned representations has various applications, such as predicting method names , detecting bugs and vulnerabilities , predicting types , detecting similar code , inferring specifications , code de-obfuscation (; a), and program repair . Many of these techniques are based on embeddings of source code, which map a given piece of code into a continuous vector representation that encodes some aspect of the semantics of the code. A core component of most code embeddings are semantic representations of identifier names, i.e., the names of variables, functions, classes, fields, etc. in source code. Similar to words in natural languages, identifiers are the basic building block of source code. Identifiers not only account for the majority of the vocabulary of source code, but they also convey important information about the (intended) meaning of code. To reason about identifiers and their meaning, code analysis techniques build on learned embeddings of identifiers, either by adapting embeddings that were originally proposed for natural languages (a; or with embeddings specifically designed for source code (a). Given the importance of identifier embeddings, a crucial challenge is measuring how effective an embedding represents the semantic relationships between identifiers. For word embeddings in natural language, the community has addressed this question through a series of gold standards (; a; ; ; ;). These gold standards define how similar two words are based on ratings by human judges, enabling an evaluation that measures how well an embedding reflects the human ratings. Unfortunately, simply reusing existing gold standards to identifiers in source code would be misleading. One reason is that the vocabularies of natural languages and source code overlap only partially, because source code contains various terms and abbreviations not found in natural language texts. Moreover, source code has a constantly growing vocabulary, as developers tend to invent new identifiers, e.g., for newly emerging application domains. Finally, even words present in both natural languages and source code may differ in their meaning due to computer science-specific meanings of some words, e.g., "float" or "string". This paper addresses the problem of measuring and comparing the effectiveness of embeddings of identifiers. We present IdBench, a benchmark for evaluating techniques that represent semantic similarities of identifiers. The basis of the benchmark is a dataset of developer opinions about the similarity of pairs of identifiers. We gather this dataset through two surveys that show realworld identifiers and code snippets to hundreds of developers, asking them to rate their similarity. Taking the developer opinions as a gold standard, IdBench allows for evaluating embeddings in a systematic way by measuring to what extent an embedding agrees with ratings given by developers. Moreover, inspecting pairs of identifiers for which an embedding strongly agrees or disagrees with the benchmark helps understand the strengths and weaknesses of current embeddings. Overall, we gather thousands of ratings from 500 developers. Cleaning and compiling this raw dataset into a benchmark yields several hundreds of pairs of identifiers with gold standard similarities, including identifiers from a wide range of application domains. We apply our approach to a corpus of JavaScript code, because several recent pieces of work on identifier names and code embeddings focus on this language (; b; a;). Applying our methodology to another language is straightforward. Based on the newly created benchmark, we evaluate and compare state-of-the-art embeddings of identifiers. We find that different embedding techniques differ heavily in terms of their ability to accurately represent identifier relatedness and similarity. The best available technique, the CBOW variant of FastText, accurately represents relatedness, but none of the available techniques accurately represents identifier similarities. One reason is that some embeddings are confused about identifiers with opposite meaning, e.g., rows and cols, and about identifiers that belong to the same application domain but are not similar. Another reason is that some embeddings miss synonyms, e.g., file and record. We also find that simple string distance functions, which measure the similarity of identifiers without any learning, are surprisingly effective, and even outperform some learned embeddings for the similarity task. In summary, this paper makes the following contributions. Methodology: To the best of our knowledge, we are the first to systematically evaluate embeddings of identifiers. Our methodology is based on surveying developers and summarizing their opinions into gold standard similarities of pairs of identifiers. Reusable benchmark: We make available a benchmark of hundreds of pairs of identifiers, providing a way to systematically evaluate existing and future embeddings. Comparison of state-of-the-art embeddings: We evaluate seven existing embeddings and string similarity functions, and discuss their strengths and weaknesses. We design IdBench so that it distinguishes two kinds of semantic relationships between identifiers (; ;). On the one hand, relatedness refers to the degree of association between two identifiers and covers various possible relations between them. For example, top and bottom are related because they are opposites, click and dblclick are related because they belong to the same general concept, and getBorderWidth and getPadding are related because they belong to the same application domain. On the other hand, similarity refers to the degree to which two identifiers have the same meaning, in the sense that one could substitute the other without changing the overall meaning. For example, length and size, as well as username and userid, are similar to each other. Similarity is a stronger semantic relationship than relatedness, because the former implies the latter, but not vice-versa. For example, the identifiers start and end are related, as they are opposites, IdBench includes three benchmark tasks: A relatedness task and two task to measure how well an embedding reflects the similarity of identifiers: a similarity task and a contextual similarity task. The following describes how we gather developer opinions that provide data for these tasks. Direct Survey of Developer Opinions This survey shows two identifiers to a developer and then directly asks how related and how similar the identifiers are. Figure 1a shows an example question from the survey. The developer is shown pairs of identifiers and is then asked to rate on a five-point Likert scale how related and how similar these identifiers are to each other. In total, each developer is shown 18 pairs of identifiers, which we randomly sample from a larger pool of pairs. Before showing the questions, we provide a brief description of what the developers are supposed to do, including an explanation of the terms "related" and "substitutable". The ratings gathered in the direct survey are the basis for the relatedness task and the similarity task of IdBench. This survey asks developers to pick an identifier that best fits a given code context, which indirectly asks about the similarity of identifiers The motivation is that identifier names alone may not provide enough information to fully judge how similar they are . For example, without any context, identifiers idx and hl may cause confusion for developers who are trying to judge their similarity. The survey addresses this challenge by showing the code context in which an identifier occurs, and by asking the developers to decide which of two given identifiers best fits this context. If, for a specific pair of identifiers, the developers choose both identifiers equally often, then the identifiers are likely to be similar to each other, since one can substitute the other. Figure 1b shows a question asked during the indirect survey. As shown in the example, for code contexts where the identifier occurs multiple times, we show multiple blanks that all refer to the same identifier. In total, we show 15 such questions to each participant of the survey. The ratings gathered in the indirect survey are the basis for the contextual similarity tasks of IdBench. We select identifiers and code contexts from a corpus of 50,000 JavaScript files . We select 300 pairs, made out of 488 identifiers, through a combination of automated and manual selection, aimed at a diverse set that covers different degrees of similarity and relatedness. The selection is guided by similarities between identifiers as judged by word embeddings Bruni et al. (2014b). The first step is to extract from the code corpus all identifier names that appear more than 100 times, which in about 17,000 identifiers, including method names, variable names, property names, other types of identifiers. Next, we compute word embeddings of the extracted identifiers, compute the cosine distance of all pairs of identifiers, and then sort the pairs according to their cosine distances. We use two word embeddings Mikolov et al. (2013a); Alon et al. (2018a), giving two sorted lists of pairs, and sample 150 pairs from each list as follows: (i) Pick 75 of the first 1,000 pairs, i.e., the most similar pairs according to the respective embedding, where 38 pairs are randomly sampled and 37 pairs are manually selected. We manually select some pairs because otherwise, we had observed a lack of pairs of synonymous identifiers. (ii) Randomly sample 50 from pairs 1,001 to 10,000, i.e., pairs that still have a relatively high similarity. (iii) Randomly sample 25 pairs from the remaining pairs, i.e., pairs that are mostly dissimilar to each other. To gather the code contexts for the indirect survey, we search the code corpus for occurrences of the selected identifiers. As the size of the context, we choose five lines, which provides sufficient context to choose the best identifier without overwhelming developers with large amounts of code. For each identifier, we randomly select five different contexts. When showing a specific pair of identifiers to a developer, we randomly select one of the gathered contexts for one of the two identifiers. Participants We pay developers via Amazon Mechanical Turk to perform the surveys. Participants take around 15 minutes to complete both surveys. In total, 500 developers participate in the survey, which yields at least 10 ratings for each pair of identifiers. To eliminate noise in the gathered data, e.g., due to lack of expertise or involvement by the participants, we clean the data by removing some participants and identifier pairs. Removing Outliers As a first filter, we remove outlier participants based on the Inter-Rater Agreement (IRA), which measures the degree of agreement between participants. We use Krippendorf's alpha coefficient, because it handles unequal sample sizes, which fits our data, since not all participants rate the same pairs, and because not all pairs have the same number of ratings. The coefficient ranges between zero and one, where zero represents complete disagreement and one represents perfect agreement. For each participant, we calculate the difference between her rating and the average of all the other ratings for each pair. Then, we average these differences for each rater, and discard participants with a difference above a threshold. We perform this computation both for the relatedness and similarity ratings from the direct survey, and then remove outliers based on the average difference across both ratings. Removing Downers As a second filter, we eliminate participants that decrease the overall IRA, called downers , because they bring the agreement level between all participants down. For each participant p, we compute IRA sim and IRA rel after removing the ratings of p from the data. If IRA sim or IRA rel increases by at least 10%, then we discard that participant's ratings. All IRA-based filtering of participants is based on ratings from the direct survey only. For the indirect survey, computing the IRA would be inadequate because the code contexts shown along with a pair of identifiers are randomly sampled, i.e., a pair is rarely shown with the same context. Instead, we exploit the fact that the two surveys are shown in sequence to each participant. Specifically, we discard participants based on the first two filters for both the direct and indirect survey, assuming that participants discarded in the direct survey are not worth keeping in the indirect survey. Removing Pairs with Confusing Contexts As a third filter, we eliminate some pairs of identifiers used in the indirect survey. Since our random selection of code contexts may include contexts that are not helpful in deciding on the most suitable identifier, the ratings for some pairs are rather arbitrary. To mitigate this problem, we remove a pair if the difference in similarity as rated in the direct and indirect surveys exceeds some threshold. Table 2 shows the number of identifier pairs that remain in the benchmark after data cleaning. For each of the three tasks, we provide a small, medium, and large benchmark, which differ by the thresholds used during data cleaning. The smaller benchmarks use stricter thresholds and hence provide higher agreements between the participants, whereas the larger benchmarks offer more pairs. Converting Ratings to Scores To ease the evaluation of embeddings, we convert the ratings gathered for a specific pair during the developer surveys into a similarity score in the range. For the direct survey, we scale the 5-point Likert-scale ratings into the range and average all ratings for . This conversion yields an unbounded distance measure d for each pair, which we convert into a similarity score s by normalizing and inverting the distance:, where min d and max d are the minimum and maximum distances across all pairs. Examples Table 3 shows representative examples of identifier pairs and their scores for the three benchmark tasks. The example illustrates that the scores match human intuition and that the gold standard clearly distinguishes relatedness from similarity. Some of the highly related and highly similar pairs, e.g., substr and substring, are lexically similar, while others are synonyms, e.g., count and total. While identifiers like rows and columns are strongly related, one cannot substitute the other, and they hence have low similarity. Similarly miny, ypos represent distinct properties of the variable y. Finally, some pairs are either weakly or not at all related, e.g., re and destruct. Embeddings and String Distance Functions To assess how well existing embedding techniques represent the relatedness and similarity of identifiers, we evaluate five vector representations against IdBench. We evaluate (i) the continuous bag of words and the skip-gram variants of Word2vec (a ;b) ("w2v-cbow" and "w2v-sg"), because recent identifier-based code analysis tools, e.g., DeepBugs and NL2Type use it, (ii) FastText , a sub-word extension of Word2vec that represents words as character n-grams ("FT-cbow" and "FT-sg"), and (iii) an embedding technique specifically designed for code, which learns from paths through a structural, tree-based representation of code (a) ("path-based"). We train all embeddings on the same code corpus of 50,000 JavaScript files. For each embedding, we experiment with various hyper-parameters (e.g., dimension, number of context words) and report only for the best performing models. In addition to neural embeddings of identifiers, we also evaluate two string distance functions: Levenshtein's edit distance and Needleman-Wunsch distance . These functions use lexical similarity as a proxy for the semantic relatedness of identifiers. We consider these functions because they are used in identifier-based code analysis tools, including a bug detection tool deployed at Google . Measuring Agreement with the Benchmark We measure the magnitude of agreement of an embedding with IdBench by computing Spearman's rank correlation ρ between the cosine similarities of pairs of identifier vectors and the gold standard of similarity scores. For string similarity functions, we compute the similarity score s = 1 − d norm for each pair based on a normalization d norm of the distance returned by the function. Figure 4 shows the agreement of the evaluated techniques with the small, medium, and large variants of IdBench. All embeddings and string distance functions agree to some extent with the gold standard. Overall, FastText-cbow consistently outperforms all other techniques, both for relatedness and similarity. We discuss the in more detail in the following. Figure 4a, all techniques achieve relatively high levels of agreement, with correlations between 46% and 73%. The neurally learned embeddings clearly outperform the string distance-based similarity functions (53-73% vs. 46-48%), showing that the effort of learning a semantic representation is worthwhile. In particular, the learned embeddings match or even slightly exceed the IRA, which is sometimes considered an upper bound of how strongly an embedding may correlate with a similarity-based benchmark . Comparing different embedding techniques with each other, we find that both FastText variants achieve higher scores than all other embeddings. In contrast, despite using additional structural information of source code, path-based embeddings score only comparably to Word2vec. Similarity Figure 4b shows a much lower agreement with the gold standard for similarity than for relatedness. One explanation is that encoding semantic similarity is a harder task than encoding the less strict notion of relatedness. Similar to relatedness, FastText-cbow shows the strongest agreement, ranging between 37% and 39%. A perhaps surprising is string distance functions outperforming some of the embeddings. Contextual similarity The of the contextual similarity task (Figure 4c), confirm the findings from the similarity task. All studied techniques are less effective than for relatedness, and FastText-cbow achieves the highest agreement with IdBench. One difference between the for similarity and contextual similarity is that we here observe higher scores by path-based embeddings. We attribute this to the ability of the path-based model to represent code snippets in a vector space. While the best available embeddings are highly effective at representing relatedness, none of the studied techniques reaches the same level of agreement for similarity. In fact, even the best in Figures 4b and 4c (39%) clearly stay beyond the IRA of our benchmark (62%), showing a huge potential for improvement. For many applications of embeddings of identifiers, semantic similarity is crucial. For example, tools to suggest suitable variable or method names (; a) aim for the name that is most similar, not only most related, to the concept represented by the variable or method. Likewise, identifier name-based tools for finding programming errors or variable misuses want to identify situations where the developer uses a wrong, but perhaps related, variable. The lack of embeddings that accurately represent the semantic similarities of identifiers motivates more work on embedding techniques suitable for this task. To better understand why current embeddings sometimes fail to accurately represent similarities, Table 1 shows the most similar identifiers of selected identifiers according to the FastText-cbow and path-based embeddings. The examples illustrate two observations. First, FastText, due to its use of n-grams , tends to cluster identifiers based on lexical commonalities. While many lexically similar identifiers are also semantically similar, e.g., substr and substring, this approach misses other synonyms, e.g., item and entry. Another downside is that lexical similarity may also establish wrong relationships. For example, substring and substrCount represent different concepts, but FastText finds them to be highly similar. Second, in contrast to FastText, path-based embeddings tend to cluster words based on their structural and syntactical contexts. This approach helps the embeddings to identify synonyms despite their lexical differences, e.g., count and total, or files and records. The downside is that it also clusters various related but not similar identifiers, e.g., minText and maxText, or substr and getPadding. Some of these identifiers even have opposing meanings, e.g., rows and cols, which can mislead code analysis tools when reasoning about the semantics of code. A somewhat surprising is that simple string distance functions achieve a level of agreement with IdBench's similarity gold standards as high as some learned embeddings. The reason why string distance functions sometimes correctly identify semantic similarities is that some semantically similar identifiers are also be lexically similar. One downside of lexical approaches is that they miss synonymous identifiers, e.g., count and total. The wide use of word embeddings in NLP raises the question of how to compare and evaluate word embeddings. Several gold standards based on human judgments have been proposed, focusing on either relatedness (; a;) or similarity (; ; ;) of words. While these existing gold standards for NL words have been extremely inspiring, they are insufficient to evaluate embeddings of identifiers. One reason is that the vocabulary of source code contains various words not found in standard NL vocabulary, e.g., abbreviations and domain-specific terms, leading to very large vocabularies . Moreover, even identifiers found also in NL may have a different meaning in source code, e.g., float or string. This work is the first to address the need for a gold standard for identifiers. Data gathering Asking human raters how related or similar two words are was first proposed by and then adopted by others (; ; ;). Our direct survey also follows this methodology. propose to gather judgments about contextual similarity by asking participants to choose a word to fill in a blank, an idea we adopt in our indirect survey. To choose words and pairs of words, prior work relies on manual selection , preexisting free association databases , e.g., USF or VerbNet (;, or cosine similarities according to pre-existing models (a; . We follow the latter approach, as it minimizes human bias while covering a wide range of degrees of relatedness and similarity. Inter-rater agreement Gold standards for NL words reach an IRA of 0.61 and 0.67 . Our "small" dataset reaches similar levels of agreement, showing that the rates in IdBench represent a genuine human intuition. As noted by , the IRA also gives an upper bound of the expected correlation between the tested model and the gold standard. Our show that current models still leave plenty of room for improvement, especially w.r.t. similarity. Embeddings of identifiers Embeddings of identifiers are at the core of several code analysis tools. A popular approach, e.g., for bug detection , type prediction , or vulnerability detection , is applying Word2vec (a; to token sequences, which corresponds to the Word2vec embedding evaluated in Section 3. train an RNN-based language model and extract its final hidden layer as an embedding of identifiers. provide a more comprehensive survey of embeddings for source code. Beyond learned embeddings, string similarity functions are used in other name-based tools, e.g., for detecting bugs or for inferring specifications . The quality of embeddings is crucial in these and other code analysis tools, and IdBench will help to improve the state of the art. Embeddings of programs Beyond embeddings of identifiers, there is a stream of work on embedding larger parts of a program. use a log-bilinear, neural language model to predict the names of methods. Other work embeds code based on graph neural networks or sequence-based neural networks applied to paths through a graph representation of code (; b; ; ; ;). Code2seq embeds code and then generates sequences of NL words (a). gives a detailed survey of learned models of code. To evaluate embeddings of programs, propose the COSET benchmark that provide thousands of programs with semantic labels. Ours and their work complement each other, as COSET evaluates embeddings of entire programs, whereas IdBench evaluates embeddings of identifiers. Since identifiers are a basic building block of source code, a benchmark for improving embeddings of identifiers will eventually also benefit learning-based code analysis tools. This paper presents the first benchmark for evaluating vector space embeddings of identifiers names, which are at the core of many machine learning models of source code. We compile thousands of ratings gathered from 500 developers into three benchmarks that provide gold standard similarity scores representing the relatedness, similarity, and contextual similarity of identifiers. Using IdBench to experimentally compare five embedding techniques and two string distance functions shows that these techniques differ significantly in their agreement with our gold standard. The best available embedding is very effective at representing how related identifiers are. However, all studied techniques show huge room for improvement in their ability to represent how similar identifiers are. IdBench will help steer future efforts on improved embeddings of identifiers, which will eventually enable better machine learning models of source code.

Represent the short summary for finding the corresponding scientific article (long)

A benchmark to evaluate neural embeddings of identifiers in source code.

Represent the TLDR for finding the corresponding scientific article

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question:
represent the text to find a very short summary (1-2 sentences max):

------

Fewshots:
'We introduce a new deep convolutional neural network, CrescendoNet, by stacking simple building blocks without residual connections. Each Crescendo block contains independent convolution paths with increased depths. The numbers of convolution layers and parameters are only increased linearly in Crescendo blocks. In experiments, CrescendoNet with only 15 layers outperforms almost all networks without residual connections on benchmark datasets, CIFAR10, CIFAR100, and SVHN. Given sufficient amount of data as in SVHN dataset, CrescendoNet with 15 layers and 4.1M parameters can match the performance of DenseNet-BC with 250 layers and 15.3M parameters. CrescendoNet provides a new way to construct high performance deep convolutional neural networks without residual connections. Moreover, through investigating the behavior and performance of subnetworks in CrescendoNet, we note that the high performance of CrescendoNet may come from its implicit ensemble behavior, which differs from the FractalNet that is also a deep convolutional neural network without residual connections. Furthermore, the independence between paths in CrescendoNet allows us to introduce a new path-wise training procedure, which can reduce the memory needed for training. Deep convolutional neural networks (CNNs) have significantly improved the performance of image classification BID3 BID25. However, training a CNN also becomes increasingly difficult with the network deepening. One of important research efforts to overcome this difficulty is to develop new neural network architectures BID6 BID14.Recently, residual network BID3 and its variants BID8 have used residual connections among layers to train very deep CNN. The residual connections promote the feature reuse, help the gradient flow, and reduce the need for massive parameters. The ResNet BID3 and DenseNet BID8 achieved state-of-the-art accuracy on benchmark datasets. Alternatively, FractalNet BID14 expanded the convolutional layers in a fractal form to generate deep CNNs. Without residual connections BID3 and manual deep supervision BID15, FractalNet achieved high performance on image classification based on network structural design only. Many studies tried to understand reasons behind the representation view of deep CNNs. BID27 showed that residual network can be seen as an ensemble of relatively shallow effective paths. However, BID2 argued that ensembles of shallow networks cannot explain the experimental of lesioning, layer dropout, and layer reshuffling on ResNet. They proposed that residual connections have led to unrolled iterative estimation in ResNet. Meanwhile, BID14 speculated that the high performance of FractalNet was due to the unrolled iterative estimation of features of the longest path using features of shorter paths. Although unrolled iterative estimation model can explain many experimental , it is unclear how it helps improve the classification performance of ResNet and FractalNet. On the other hand, the ensemble model can explain the performance improvement easily. In this work, we propose CrescendoNet, a new deep convolutional neural network with ensemble behavior. Same as other deep CNNs, CrescendoNet is created by stacking simple building blocks, called Crescendo blocks FIG0 ). Each Crescendo block comprises a set of independent feed-forward paths with increased number of convolution and batch-norm layers (a). We only use the identical size, 3 × 3, for all convolutional filters in the entire network. Despite its simplicity, CrescendoNet shows competitive performance on benchmark CIFAR10, CI-FAR100, and SVHN datasets. Similar to FractalNet, CrescendoNet does not include residual connections. The high performance of CrescendoNet also comes completely from its network structural design. Unlike the FractalNet, in which the numbers of convolutional layers and associated parameters are increased exponentially, the numbers of convolutional layers and parameters in Crescendo blocks are increased linearly. CrescendoNet shows clear ensemble behavior (Section 3.4). In CrescendoNet, although the longer paths have better performances than those of shorter paths, the combination of different length paths have even better performance. A set of paths generally outperform its subsets. This is different from FractalNet, in which the longest path alone achieves the similar performance as the entire network does, far better than other paths do. Furthermore, the independence between paths in CrescendoNet allows us to introduce a new pathwise training procedure, in which paths in each building block are trained independently and sequentially. The path-wise procedure can reduce the memory needed for training. Especially, we can reduce the amortized memory used for training CrescendoNet to about one fourth. We summarize our contribution as follows:• We propose the Crescendo block with linearly increased convolutional and batch-norm layers. The CrescendoNet generated by stacking Crescendo blocks further demonstrates that the high performance of deep CNNs can be achieved without explicit residual learning.• Through our analysis and experiments, we discovered an emergent behavior which is significantly different from which of FractalNet. The entire CrescendoNet outperforms any subset of it can provide an insight of improving the model performance by increasing the number of paths by a pattern.• We introduce a path-wise training approach for CrescendoNet, which can lower the memory requirements without significant loss of accuracy given sufficient data. Crescendo Block The Crescendo block is built by two layers, the convolution layer with the activation function and the following batch normalization layer BID10. The convolutional layers have the identical size, 3 × 3. The Conv-Activation-BatchNorm unit f 1, defined in the Eq.1 is the base branch of the Crescendo block. We use ReLU as the activation function to avoid the problem of vanishing gradients BID17. DISPLAYFORM0 The variable z denotes the input feature maps. We use two hyper-parameters, the scale S and the interval I to define the structure of the Crescendo block H S. The interval I specifies the depth difference between every two adjacent branches and the scale S sets the number of branches per block. The structure of the n th branch is defined by the following equation: DISPLAYFORM1 where the superscript nI is the number of recursion time of the function f 1. The structure of Crescendo block H S can be obtained below: DISPLAYFORM2 where ⊕ denotes an element-wise averaging operation. Note that the feature maps from each path are averaged element-wise, leaving the width of the channel unchanged. A Crescendo block with S = 4 and I = 1 is shown in FIG0.The structure of Crescendo block is designed for exploiting more feature expressiveness. The different depths of parallel paths lead to different receptive fields and therefore generate features in different abstract levels. In addition, such an incremental and parallel form explicitly supports the ensemble effects, which shows excellent characteristics for efficient training and anytime classification. We will explain and demonstrate this in the following sections. CrescendoNet Architecture The main body of CrescendoNet is composed of stacked Crescendo blocks with max-pooling layers between adjacent blocks (FIG0). Following the main body, like most deep CNNs, we use two fully connected layers and a soft-max layer as the classifier. In all experiments, the two fully connected layers have 384 hidden units and 192 hidden units respectively. The overall structure of CrescendoNet is simple and we only need to tune the Crescendo block to modify the entire network. To reduce the memory consumption during training CrescendoNet, we propose a path-wise training procedure, leveraging the independent multi-path structure of our model. We denote stacked Conv-BatchNorm layers in one Crescendo block as one path. We train each path individually, from the shortest to the longest repetitively. When we are training one path, we freeze the parameters of other paths. In other words, these frozen layers only provide learned features to support the training. FIG1 illustrates the procedure of path-wise training within a CrescendoNet block containing four paths. There are two advantages of path-wise training. First, path-wise training procedure significantly reduces the memory requirements for convolutional layers, which constitutes the major memory cost for training CNNs. For example, the higher bound of the memory required for computation and storage of gradients using momentum stochastic gradient descent algorithms can be reduced to about 40% for a Crescendo block with 4 paths where interval = 1. Second, path-wise training works well with various optimizers and regularizations. Even dropout and drop-path can be applied to the model during the training. Dropout and drop-connect BID28, which randomly set a selected subset of activations or weights to zero respectively, are effective regularization techniques for deep neural networks. Their variant, drop-path BID14, shows further performance improvement by dropping paths when training FractalNet. We use both dropout and drop-path for regularizing the Crescendo block. We drop the branches in each block with a predefined probability. For example, given drop-path rate, p = 0.3, the expectation of the number of dropped branches is 1.2 for a Crescendo block with four branches. For the fully connected layers, we use L2 norm of their weights as an additional term to the loss. We evaluate our models with three benchmark datasets: CIFAR10, CIFAR100, and Street View House Numbers (SVHN) BID19. CIFAR10 and CIFAR100 each have 50,000 training images and 10,000 test images, belonging to 10 and 100 classes respectively. All the images are in RGB format with the size of 32 × 32-pixel. SVHN are color images, with the same size of 32 × 32-pixel, containing 604,388 and 26,032 images for training and testing respectively. Note that these digits are cropped from a series of numbers. Thus, there may be more than one digit in an image, but only the one in the center is used as the label. For data augmentation, we use a widely adopted scheme BID16 BID14 BID6 b; BID24 BID21 BID3. We first pad images with 4 zero pixels on each side, then crop padded images to 32 × 32-pixel randomly and horizontally flipping with a 50% probability. We preprocess each image in all three datasets by subtracting off the mean and dividing the variance of the pixels. We use Mini-batch gradient descent to train all our models. We implement our models using TensorFlow distributed computation framework BID0 and run them on NVidia P100 GPU. We also optimize our models by adaptive momentum estimation (Adam) optimization BID11 and Nesterov Momentum optimization BID18 respectively. For Adam optimization, we set the learning rate hyper-parameter to 0.001 and let Adam adaptively tune the learning rate during the training. We choose the momentum decay hyper-parameter β 1 = 0.9 and β 2 = 0.999. And we set the smoothing term = 10 −8. This configuration is the default setting for the AdamOptimizer class in TensorFlow. For Nesterov Momentum optimization, we set the hyper-parameter momentum = 0.9. We decay the learning rate from 0.1 to 0.01 after 512 epochs for CIFAR and from 0.05 to 0.005, then to 0.0005, after 42 epochs and 63 epochs respectively for SVHN. We use truncated normal distribution for parameter initialization. The standard deviation of hyper-parameters is 0.05 for convolutional weights and 0.04 for fully connected layer weights. For all datasets, we use the batch size of 128 on each training replica. For the whole net training, we run 700 epochs on CIFAR and 70 epochs on SVHN. For the path-wise training, we run 1400 epochs on CIFAR and 100 epochs on SVHN.Using a CrescendoNet model with three blocks each contains four branches as illustrated in FIG0, we investigate the following preliminary aspects: the model performance under different block widths, the ensemble effect, and the path-wise training performance. We study the Crescendo block with three different width configurations: equal width globally, equal width within the block, and increasing width. All the three configurations have the same fully connected layers. For the first one, we set the number of feature maps to 128 for all the convolutional layers. For the second, the numbers of feature maps are for convolutional layers in each block. For the last, we gradually increase the feature maps for each branch in three blocks to correspondingly. For example, the number of feature maps for the second and fourth branches in the second block is and. The exact number of maps for each layer is defined by the following equation: DISPLAYFORM0 where n maps denotes the number of feature maps for a layer, n inmaps and n outmaps are number of input and output maps respectively, n layers is the number of layers in the block, and i layer is the index of the layer in the branch, starting from 1.To inspect the ensemble behavior of CrescendoNet, we compare the performance of models with and without drop-path technique and subnets composed by different combinations of branches in each block. For the simplicity, we denote the branch combination as a set P containing the index of the branch. For example, P = {1, 3} means the blocks in the subnet only contains the first and third branches. The same notation is used in TAB0 and FIG2. Table 1 gives a comparison among CrescendoNet and other representative models on CIFAR and SVHN benchmark datasets. For five datasets, CrescendoNet with only 15 layers outperforms almost all networks without residual connections, plus original ResNet and ResNet with Stochastic Depth. For CIFAR10 and CIFAR100 without data augmentation, CrescendoNet also performs better than all the given models except DenseNet with bottleneck layers and compression (DenseNet-BC) with 250 layers. However, CrescendoNet's error rate 1.76% matches the 1.74% error rate of given DenseNet-BC, on SVHN dataset which has plentiful data for each class. Comparing with FractalNet, another outstanding model without residual connection, CrescendoNet has a simpler structure, fewer parameters, but higher accuracies. The lower rows in Table 1 compare the performance of our model given different configuration. In three different widths, the performance simultaneously grows with the number of feature maps. In other words, there is no over-fitting when we increase the capacity of CrescendoNet in an appropriate scope. Thus, CrescendoNet demonstrates a potential to further improve its performance by scaling up. In addition, the drop-path technique shows its benefits to our models on all the datasets, just as it does to FractalNet. Another interesting from Table 1 is the performance comparison between Adam and Nesterov Momentum optimization methods. Comparing with Nesterov Momentum method, Adam performs similarly on CIFAR10 and SVHN, but worse on CIFAR100. Note that there are roughly 60000, 5000, and 500 training images for each class in SVHN, CIFAR10, and CIFAR100 respectively. This implies that Adam may be a better option for training CrescendoNet when the training data is abundant, due to the convenience of its adaptive learning rate scheduling. The last row of Table 1 gives the from path-wise training. Training the model with less memory requirement can be achieved at the cost of some performance degradation. However, Pathwise trained CrescendoNet still outperform many of networks without residual connections on given datasets.. The show the ensemble behavior of our model. Specifically, the more paths contained in the network, the better the Table 1: Whole net classification error (%) on CIFAR10/CIFAR100/SVHN. We highlight the top three accuracies in each column with the bold font. The three numbers in the parentheses denote the number of output feature maps of each block. The plus sign (+) denotes the data augmentation. The sign (-W) means that the feature maps of layers in each branch increase as explained in the model configuration section. The compared models include: Network in Network BID24, ALL-CNN , Deeply Supervised Net BID15, Highway Network BID24, FractalNet BID14, ResNet BID3, ResNet with Stochastic Depth BID7, Wide ResNet BID30, and DenseNet BID6 performance. And the whole net outperforms any single path network with a large margin. For example, the whole net and the net based on the longest path show the inference error rate of 6.90% and 10.69% respectively, for CIFAR10 without data augmentation. This implicit ensemble behavior differentiates CrescendoNet from FractalNet, which shows a student-teacher effect. Specifically, the longest path in FractalNet can achieve a similar or even lower error rate compared to the whole net. To investigate the dynamic behavior of subnets, we test the error rate changes of subnets during the training. We use Adam to train the CrescendoNet with the structure shown in FIG0 on CIFAR10 for 450 epochs. FIG2 illustrates the behavior of different path combinations during the training. It shows that the inference accuracy of the whole net grows simultaneously with all the subnets, which demonstrates the ensemble effect. Second, for any single path network, the performance grows with the depth. This behavior of the anytime classifier is also shown by FractalNet. In other words, we could use the short path network to give a rough but quick inference, then use more paths to gradually increase the accuracy. This may be useful for time-critical applications, like integrated recognition system for autonomous driving tasks. Conventional deep CNNs, such as AlexNet VGG-19 (Simonyan & BID20, directly stacked the convolutional layers. However, the vanishing gradient problem makes it difficult to train and tune very deep CNN of conventional structures. Recently, stacking small convolutional blocks has become an important method to build deep CNNs. Introducing new building blocks becomes the key to improve the performance of deep CNN. BID16 first introduced the NetworkInNetwork module which is a micro neural network using a multiple layer perceptron (MLP) for local modeling. Then, they piled the micro neural networks into a deep macro neural network. BID25 introduced a new building block called Inception, based on which they built GoogLeNet. Each Inception block has four branches of shallow CNNs, building by convolutional kernels with size 1 × 1, 3 × 3, 5 × 5, and max-pooling with kernel size 3 × 3. Such a multiple-branch scheme is used to extract diversified features while reducing the need for tuning the convolutional sizes. The main body of GoogLeNet has 9 Inception blocks stacked each other. Stacking multiplebranch blocks can create an exponential combination of feed-forward paths. Such a structure com-bined with the dropout technique can show an implicit ensemble effect BID27 BID22. GoogLeNet was further improved with new blocks to more powerful models, such as Xception BID1 and Inception-v4 BID26. To improve the scalability of GoogLeNet, BID26 used convolution factorization and label-smoothing regularization in Inception-v4. In addition, BID1 explicitly defined a depth-wise separable convolution module replacing Inception module. Recently, BID14 introduced FractalNet built by stacked Fractal blocks, which are the combination of identical convolutional layers in a fractal expansion fashion. FractalNet showed that it is possible to train very deep neural network through the network architecture design. FractalNet implicitly also achieved deep supervision and student-teacher learning by the fractal architecture. However, the fractal expansion form increases the number of convolution layers and associated parameters exponentially. For example, the original FractalNet model with 21 layers has 38.6 million parameters, while a ResNet of depth 1001 with similar accuracy has only 10.2 million parameters BID6. Thus, the exponential expansion reduced the scalability of FractalNet. Another successful idea in network architecture design is the use of skip-connections BID3 b; BID6 BID30 BID29. ResNet BID3 used the identity mapping to short connect stacked convolutional layers, which allows the data to pass from a layer to its subsequent layers. With the identity mapping, it is possible to train a 1000-layer convolutional neural network. BID6 recently proposed DenseNet with extremely residual connections. They connected each layer in the Dense block to every subsequent layer. DenseNet achieved the best performance on benchmark datasets so far. On the other hand, Highway networks BID23 used skip-connections to adaptively infuse the input and output of traditional stacked neural network layers. Highway networks have helped to achieve high performance in language modeling and translation. CNN has shown excellent performance on image recognition tasks. However, it is still challenging to tune, modify, and design an CNN. We propose CrescendoNet, which has a simple convolutional neural network architecture without residual connections BID3. Crescendo block uses convolutional layers with same size 3 × 3 and joins feature maps from each branch by the averaging operation. The number of convolutional layers grows linearly in CrescendoNet while exponentially in FractalNet BID14. This leads to a significant reduction of computational complexity. Even with much fewer layers and a simpler structure, CrescendoNet matches the performance of the original and most of the variants of ResNet on CIFAR10 and CIFAR100 classification tasks. Like FractalNet BID14, we use dropout and drop-path as regularization mechanisms, which can train CrescendoNet to be an anytime classifier, namely, CrescendoNet can perform inference with any combination of the branches according to the latency requirements. Our experiments also demonstrated that CrescendoNet synergized well with Adam optimization, especially when the training data is sufficient. In other words, we can avoid scheduling the learning rate which is usually performed empirically for training existing CNN architectures. CrescendoNet shows a different behavior from FractalNet in experiments on CIFAR10/100 and SVHN. In FractalNet BID14, the longest path alone achieves the similar performance as the entire network, far better than other paths, which shows the student-teacher effect. The whole FractalNet except the longest path acts as a scaffold for the training and becomes dispensable later. On the other hand, CrescendoNet shows that the whole network significantly outperforms any set of it. This fact sheds the light on exploring the mechanism which can improve the performance of deep CNNs by increasing the number of paths.' == 'We introduce CrescendoNet, a deep CNN architecture by stacking simple building blocks without residual connections.' != 'We propose accelerating Batch Normalization (BN) through sampling less correlated data for reduction operations  with regular execution pattern, which achieves up to 2x and 20% speedup for BN itself and the overall training, respectively.'

Question: Generative adversarial nets (GANs) are widely used to learn the data sampling process and their performance may heavily depend on the loss functions, given a limited computational budget. This study revisits MMD-GAN that uses the maximum mean discrepancy (MMD) as the loss function for GAN and makes two contributions. First, we argue that the existing MMD loss function may discourage the learning of fine details in data as it attempts to contract the discriminator outputs of real data. To address this issue, we propose a repulsive loss function to actively learn the difference among the real data by simply rearranging the terms in MMD. Second, inspired by the hinge loss, we propose a bounded Gaussian kernel to stabilize the training of MMD-GAN with the repulsive loss function. The proposed methods are applied to the unsupervised image generation tasks on CIFAR-10, STL-10, CelebA, and LSUN bedroom datasets. Results show that the repulsive loss function significantly improves over the MMD loss at no additional computational cost and outperforms other representative loss functions. The proposed methods achieve an FID score of 16.21 on the CIFAR-10 dataset using a single DCGAN network and spectral normalization. Generative adversarial nets (GANs) BID7 ) are a branch of generative models that learns to mimic the real data generating process. GANs have been intensively studied in recent years, with a variety of successful applications (; Li et al. (2017b);;; BID13 ). The idea of GANs is to jointly train a generator network that attempts to produce artificial samples, and a discriminator network or critic that distinguishes the generated samples from the real ones. Compared to maximum likelihood based methods, GANs tend to produce samples with sharper and more vivid details but require more efforts to train. Recent studies on improving GAN training have mainly focused on designing loss functions, network architectures and training procedures. The loss function, or simply loss, defines quantitatively the difference of discriminator outputs between real and generated samples. The gradients of loss functions are used to train the generator and discriminator. This study focuses on a loss function called maximum mean discrepancy (MMD), which is well known as the distance metric between two probability distributions and widely applied in kernel two-sample test BID8 ). Theoretically, MMD reaches its global minimum zero if and only if the two distributions are equal. Thus, MMD has been applied to compare the generated samples to real ones directly (; BID5) and extended as the loss function to the GAN framework recently (; Li et al. (2017a); ).In this paper, we interpret the optimization of MMD loss by the discriminator as a combination of attraction and repulsion processes, similar to that of linear discriminant analysis. We argue that the existing MMD loss may discourage the learning of fine details in data, as the discriminator attempts to minimize the within-group variance of its outputs for the real data. To address this issue, we propose a repulsive loss for the discriminator that explicitly explores the differences among real data. The proposed loss achieved significant improvements over the MMD loss on image generation tasks of four benchmark datasets, without incurring any additional computational cost. Furthermore, a bounded Gaussian kernel is proposed to stabilize the training of discriminator. As such, using a single kernel in MMD-GAN is sufficient, in contrast to a linear combination of kernels used in Li et al. (2017a) and. By using a single kernel, the computational cost of the MMD loss can potentially be reduced in a variety of applications. The paper is organized as follows. Section 2 reviews the GANs trained using the MMD loss (MMD-GAN). We propose the repulsive loss for discriminator in Section 3, introduce two practical techniques to stabilize the training process in Section 4, and present the of extensive experiments in Section 5. In the last section, we discuss the connections between our model and existing work. In this section, we introduce the GAN model and MMD loss. Consider a random variable X ∈ X with an empirical data distribution P X to be learned. A typical GAN model consists of two neural networks: a generator G and a discriminator D. The generator G maps a latent code z with a fixed distribution P Z (e.g., Gaussian) to the data space X: y = G(z) ∈ X, where y represents the generated samples with distribution P G. The discriminator D evaluates the scores D(a) ∈ R d of a real or generated sample a. This study focuses on image generation tasks using convolutional neural networks (CNN) for both G and D.Several loss functions have been proposed to quantify the difference of the scores between real and generated samples: {D(x)} and {D(y)}, including the minimax loss and non-saturating loss BID7 ), hinge loss , Wasserstein loss; BID10 ) and maximum mean discrepancy (MMD) (Li et al. (2017a); ) (see Appendix B.1 for more details). Among them, MMD uses kernel embedding φ(a) = k(·, a) associated with a characteristic kernel k such that φ is infinite-dimensional and φ(a), φ(b) H = k(a, b). The squared MMD distance between two distributions P and Q is DISPLAYFORM0 The kernel k(a, b) measures the similarity between two samples a and b. BID8 proved that, using a characteristic kernel k, M,P G ) reaches its minimum if and only if P X = P G (Li et al. (2017a) ). Thus, the objective functions for G and D could be (Li et al. (2017a); ): min DISPLAYFORM1 ) MMD-GAN has been shown to be more effective than the model that directly uses MMD as the loss function for the generator G (Li et al. (2017a)). showed that MMD and Wasserstein metric are weaker objective functions for GAN than the Jensen-Shannon (JS) divergence (related to minimax loss) and total variation (TV) distance (related to hinge loss). The reason is that convergence of P G to P X in JS-divergence and TV distance also implies convergence in MMD and Wasserstein metric. Weak metrics are desirable as they provide more information on adjusting the model to fit the data distribution . proved that the GAN trained using the minimax loss and gradient updates on model parameters is locally exponentially stable near equilibrium, while the GAN using Wasserstein loss is not. In Appendix A, we demonstrate that the MMD-GAN trained by gradient descent is locally exponentially stable near equilibrium. In this section, we interpret the training of MMD-GAN (using L First, consider a linear discriminant analysis (LDA) model as the discriminator. The task is to find a projection w to maximize the between-group variance w T µ x − w T µ y and minimize the withingroup variance w T (Σ x + Σ y)w, where µ and Σ are group mean and covariance. In MMD-GAN, the neural-network discriminator works in a similar way as LDA. By minimizing L att D, the discriminator D tackles two tasks: DISPLAYFORM0, causes the two groups {D(x)} and {D(y)} to repel each other (see FIG1, or maximize betweengroup variance; and 2) D increases DISPLAYFORM1 e. contracts {D(x)} and {D(y)} within each group (see FIG1, or minimize the within-group variance. We refer to loss functions that contract real data scores as attractive losses. We argue that the attractive loss L att D (Eq. 3) has two issues that may slow down the GAN training:1. The discriminator D may focus more on the similarities among real samples (in order to contract {D(x)}) than the fine details that separate them. Initially, G produces low-quality samples and it may be adequate for D to learn the common features of {x} in order to distinguish between {x} and {y}. Only when {D(y)} is sufficiently close to {D(x)} will D learn the fine details of {x} to be able to separate {D(x)} from {D(y)}. Consequently, D may leave out some fine details in real samples, thus G has no access to them during training.2. As shown in FIG1, the gradients on D(y) from the attraction (blue arrows) and repulsion (orange arrows) terms in L att D (and thus L mmd G) may have opposite directions during training. Their summation may be small in magnitude even when D(y) is far away from D(x), which may cause G to stagnate locally. Therefore, we propose a repulsive loss for D to encourage repulsion of the real data scores {D(x)}: DISPLAYFORM2 The generator G uses the same MMD loss L mmd G as before (see Eq. 2). Thus, the adversary lies in the fact that D contracts {D(y)} via maximizing FIG1 ) while G expands {D(y)} (see FIG1). Additionally, D also learns to separate the real data by minimizing DISPLAYFORM3 DISPLAYFORM4, which actively explores the fine details in real samples and may in more meaningful gradients for G. Note that in Eq. 4, D does not explicitly push the average score of {D(y)} away from that of {D(x)} because it may have no effect on the pair-wise sample distances. But G aims to match the average scores of both groups. Thus, we believe, compared to the model using DISPLAYFORM5 and L rep D is less likely to yield opposite gradients when {D(y)} and {D(x)} are distinct (see FIG1). In Appendix A, we demonstrate that GAN trained using gradient descent and the repulsive MMD loss (L At last, we identify a general form of loss function for the discriminator D: DISPLAYFORM6 In this section, we propose two approaches to stabilize the training of MMD-GAN: 1) a bounded kernel to avoid the saturation issue caused by an over-confident discriminator; and 2) a generalized power iteration method to estimate the spectral norm of a convolutional kernel, which was used in spectral normalization on the discriminator in all experiments in this study unless specified otherwise. For MMD-GAN, the following two kernels have been used:• Gaussian radial basis function (RBF), or Gaussian kernel (Li et al. (2017a) DISPLAYFORM0 2 ) where σ > 0 is the kernel scale or bandwidth.• Rational quadratic kernel DISPLAYFORM1, where the kernel scale α > 0 corresponds to a mixture of Gaussian kernels with a Gamma(α, 1) prior on the inverse kernel scales σ −1.It is interesting that both studies used a linear combination of kernels with five different kernel scales, i.e., DISPLAYFORM2, where σ i ∈ {1, 2, 4, 8, 16}, α i ∈ {0.2, 0.5, 1, 2, 5} (see FIG0 and 2c for illustration). We suspect the reason is that a single kernel k(a, b) is saturated when the distance a − b is either too large or too small compared to the kernel scale (see FIG0 and 2d), which may cause diminishing gradients during training. Both Li et al. (2017a) and applied penalties on the discriminator parameters but not to the MMD loss itself. Thus the saturation issue may still exist. Using a linear combination of kernels with different kernel scales may alleviate this issue but not eradicate it. Inspired by the hinge loss (see Appendix B.1), we propose a bounded RBF (RBF-B) kernel for the discriminator. The idea is to prevent D from pushing {D(x)} too far away from {D(y)} and causing saturation. For L att D in Eq. 3, the RBF-B kernel is: DISPLAYFORM3 For L rep D in Eq. 4, the RBF-B kernel is: DISPLAYFORM4 where b l and b u are the lower and upper bounds. As such, a single kernel is sufficient and we set σ = 1, b l = 0.25 and b u = 4 in all experiments for simplicity and leave their tuning for future work. It should be noted that, like the case of hinge loss, the RBF-B kernel is used only for the discriminator to prevent it from being over-confident. The generator is always trained using the original RBF kernel, thus we retain the interpretation of MMD loss L mmd G as a metric. RBF-B kernel is among many methods to address the saturation issue and stabilize MMD-GAN training. We found random sampling kernel scale, instance noise (Sønderby et al. ) and label smoothing may also improve the model performance and stability. However, the computational cost of RBF-B kernel is relatively low. Without any Lipschitz constraints, the discriminator D may simply increase the magnitude of its outputs to minimize the discriminator loss, causing unstable training 3. Spectral normalization divides the weight matrix of each layer by its spectral norm, which imposes an upper bound on the magnitudes of outputs and gradients at each layer of D . However, to estimate the spectral norm of a convolution kernel, reshaped the kernel into a matrix. We propose a generalized power iteration method to directly estimate the spectral norm of a convolution kernel (see Appendix C for details) and applied spectral normalization to the discriminator in all experiments. In Appendix D.1, we explore using gradient penalty to impose the Lipschitz constraint BID10;; ) for the proposed repulsive loss. In this section, we empirically evaluate the proposed 1) repulsive loss L (Li et al. (2017a) ) and rational quadratic kernel (MMD-rq) ), as well as non-saturating loss BID7 ) and hinge loss . To show the efficacy of RBF-B kernel, we applied it to both L ). DISPLAYFORM0 Dataset: The loss functions were evaluated on four datasets: 1) CIFAR-10 (50K images, 32 × 32 pixels) ; 2) STL-10 (100K images, 48 × 48 pixels) BID3 ); 3) CelebA (about 203K images, 64 × 64 pixels) (Liu et al. FORMULA8); and 4) LSUN bedrooms (around 3 million images, 64×64 pixels) (Yu et al. FORMULA8). The images were scaled to range [−1, 1] to avoid numeric issues. FORMULA8 ) was used in the generator, and spectral normalization with the generalized power iteration (see Appendix C) in the discriminator. For MMD related losses, the dimension of discriminator output layer was set to 16; for non-saturating loss and hinge loss, it was 1. In Appendix D.2, we investigate the impact of discriminator output dimension on the performance of repulsive loss. ) and thus omitted.3 On LSUN-bedroom, MMD-rbf and MMD-rq did not achieve reasonable and thus are omitted. Hyper-parameters: We used Adam optimizer (Kingma & Ba FORMULA8) with momentum parameters β 1 = 0.5, β 2 = 0.999; two-timescale update rule (TTUR) BID12 ) with two learning rates (ρ D, ρ G) chosen from {1e-4, 2e-4, 5e-4, 1e-3} (16 combinations in total); and batch size 64. Fine-tuning on learning rates may improve the model performance, but constant learning rates were used for simplicity. All models were trained for 100K iterations on CIFAR-10, STL-10, CelebA and LSUN bedroom datasets, with n dis = 1, i.e., one discriminator update per generator update 4. For MMD-rbf, the kernel scales σ i ∈ {1, √ 2, 2, 2 √ 2, 4} were used due to a better performance than the original values used in Li et al. (2017a). For MMD-rq, α i ∈ {0.2, 0.5, 1, 2, 5}. For MMD-rbf-b, MMD-rep, MMD-rep-b, a single Gaussian kernel with σ = 1 was used. Evaluation metrics: Inception score (IS) , Fréchet Inception distance (FID) BID12 ) and multi-scale structural similarity (MS-SSIM) were used for quantitative evaluation. Both IS and FID are calculated using a pre-trained Inception model . Higher IS and lower FID scores indicate better image quality. MS-SSIM calculates the pair-wise image similarity and is used to detect mode collapses among images of the same class . Lower MS-SSIM values indicate perceptually more diverse images. For each model, 50K randomly generated samples and 50K real samples were used to calculate IS, FID and MS-SSIM. TAB0 shows the Inception score, FID and MS-SSIM of applying different loss functions on the benchmark datasets with the optimal learning rate combinations tested experimentally. Note that the same training setup (i.e., DCGAN + BN + SN + TTUR) was applied for each loss function. We observed that: 1) MMD-rep and MMD-rep-b performed significantly better than MMD-rbf and MMD-rbf-b respectively, showing the proposed repulsive loss L rep D (Eq. 4) greatly improved over the attractive loss L att D (Eq. 3); 2) Using a single kernel, MMD-rbf-b performed better than MMD-rbf and MMD-rq which used a linear combination of five kernels, indicating that the kernel saturation may be an issue that slows down MMD-GAN training; 3) MMD-rep-b performed comparable or better than MMD-rep on benchmark datasets where we found the RBF-B kernel managed to stabilize MMD-GAN training using repulsive loss. 4) MMD-rep and MMD-rep-b performed significantly better than the non-saturating and hinge losses, showing the efficacy of the proposed repulsive loss. Additionally, we trained MMD-GAN using the general loss L D,λ (Eq. 5) for discriminator and L mmd G (Eq. 2) for generator on the CIFAR-10 dataset. Each color bar represents the FID score using a learning rate combination (ρ D, ρ G), in the order of (1e-4, 1e-4), (1e-4, 2e-4),...,(1e-3, 1e-3). The discriminator was trained using L D,λ (Eq. 5) with λ ∈ {-1, -0.5, 0, 0.5, 1, 2}, and generator using L mmd G (Eq. 2). We use the FID> 30 to indicate that the model diverged or produced poor .of MMD-GAN with RBF and RBF-B kernel 5. Note that when λ = −1, the models are essentially MMD-rbf (with a single Gaussian kernel) and MMD-rbf-b when RBF and RBF-B kernel are used respectively. We observed that: 1) the model performed well using repulsive loss (i.e., λ ≥ 0), with λ = 0.5, 1 slightly better than λ = −0.5, 0, 2; 2) the MMD-rbf model can be significantly improved by simply increasing λ from −1 to −0.5, which reduces the attraction of discriminator on real sample scores; 3) larger λ may lead to more diverged models, possibly because the discriminator focuses more on expanding the real sample scores over adversarial learning; note when λ 1, the model would simply learn to expand all real sample scores and pull the generated sample scores to real samples', which is a divergent process; 4) the RBF-B kernel managed to stabilize MMD-rep for most diverged cases but may occasionally cause the FID score to rise up. The proposed methods were further evaluated in Appendix A, C and D. In Appendix A.2, we used a simulation study to show the local stability of MMD-rep trained by gradient descent, while its global stability is not guaranteed as bad initialization may lead to trivial solutions. The problem may be alleviated by adjusting the learning rate for generator. In Appendix C.3, we showed the proposed generalized power iteration (Section 4.2) imposes a stronger Lipschitz constraint than the method in , and benefited MMD-GAN training using the repulsive loss. Moreover, the RBF-B kernel managed to stabilize the MMD-GAN training for various configurations of the spectral normalization method. In Appendix D.1, we showed the gradient penalty can also be used with the repulsive loss. In Appendix D.2, we showed that it was better to use more than one neuron at the discriminator output layer for the repulsive loss. The discriminator outputs may be interpreted as a learned representation of the input samples. FIG8 visualizes the discriminator outputs learned by the MMD-rbf and proposed MMD-rep methods on CIFAR-10 dataset using t-SNE (van der Maaten FORMULA4). MMD-rbf ignored the class structure in data (see FIG8) while MMD-rep learned to concentrate the data from the same class and separate different classes to some extent FIG8. This is because the discriminator D has to actively learn the data structure in order to expands the real sample scores {D(x)}. Thus, we speculate that techniques reinforcing the learning of cluster structures in data may further improve the training of MMD-GAN.In addition, the performance gain of proposed repulsive loss (Eq. 4) over the attractive loss (Eq. 3) comes at no additional computational cost. In fact, by using a single kernel rather than a linear combination of kernels, MMD-rep and MMD-rep-b are simpler than MMD-rbf and MMD-rq. Besides, given a typically small batch size and a small number of discriminator output neurons (64 and 16 in our experiments), the cost of MMD over the non-saturating and hinge loss is marginal compared to the convolution operations. In Appendix D.3, we provide some random samples generated by the methods in our study. This study extends the previous work on MMD-GAN (Li et al. (2017a) ) with two contributions. First, we interpreted the optimization of MMD loss as a combination of attraction and repulsion processes, and proposed a repulsive loss for the discriminator that actively learns the difference among real data. Second, we proposed a bounded Gaussian RBF (RBF-B) kernel to address the saturation issue. Empirically, we observed that the repulsive loss may in unstable training, due to factors including initialization (Appendix A.2), learning rate (FIG7 and Lipschitz constraints on the discriminator (Appendix C.3). The RBF-B kernel managed to stabilize the MMD-GAN training in many cases. Tuning the hyper-parameters in RBF-B kernel or using other regularization methods may further improve our . The theoretical advantages of MMD-GAN require the discriminator to be injective. The proposed repulsive loss (Eq. 4) attempts to realize this by explicitly maximizing the pair-wise distances among the real samples. Li et al. (2017a) achieved the injection property by using the discriminator as the encoder and an auxiliary network as the decoder to reconstruct the real and generated samples, which is more computationally extensive than our proposed approach. On the other hand,; imposed a Lipschitz constraint on the discriminator in MMD-GAN via gradient penalty, which may not necessarily promote an injective discriminator. The idea of repulsion on real sample scores is in line with existing studies. It has been widely accepted that the quality of generated samples can be significantly improved by integrating labels (; ;) or even pseudo-labels generated by k-means method BID9 ) in the training of discriminator. The reason may be that the labels help concentrate the data from the same class and separate those from different classes. Using a pre-trained classifier may also help produce vivid image samples BID14 ) as the learned representations of the real samples in the hidden layers of the classifier tend to be well separated/organized and may produce more meaningful gradients to the generator. At last, we note that the proposed repulsive loss is orthogonal to the GAN studies on designing network structures and training procedures, and thus may be combined with a variety of novel techniques. For example, the ResNet architecture BID11 ) has been reported to outperform the plain DCGAN used in our experiments on image generation tasks (; BID10) and self-attention module may further improve the . On the other hand, proposed to progressively grows the size of both discriminator and generator and achieved the state-of-the-art performance on unsupervised training of GANs on the CIFAR-10 dataset. Future work may explore these directions. This section demonstrates that, under mild assumptions, MMD-GAN trained by gradient descent is locally exponentially stable at equilibrium. It is organized as follows. The main assumption and proposition are presented in Section A.1, followed by simulation study in Section A.2 and proof in Section A.3. We discuss the indications of assumptions on the discriminator of GAN in Section A.4. We consider GAN trained using the MMD loss L DISPLAYFORM0 where Thus in contrast to Assumption 1, we assume Assumption 2. For GANs using MMD loss in Eq. S1, and random initialization on parameters, at equilibrium, DISPLAYFORM1 DISPLAYFORM2 is not constant almost everywhere. We use a simulation study in Section A.2 to show that D θ * D (x) = 0 does not hold in general for MMD loss. Based on Assumption 2, we propose the following proposition and prove it in Appendix A.3: Proposition 1. If there exists θ * G ∈ Θ G such that P θ * G = P X, then GANs with MMD loss in Eq. S1 has equilibria (θ * G, θ D) for any θ D ∈ Θ D. Moreover, the model trained using gradient descent methods is locally exponentially stable at (θ * DISPLAYFORM3 There may exist non-realizable cases where the mapping between P Z and P X cannot be represented by any generator G θ G with θ G ∈ Θ G . In Section A.2, we use a simulation study to show that both the attractive MMD loss L att D (Eq. S1b) and the proposed repulsive loss L rep D (Eq. S1c) may be locally stable and leave the proof for future work. In this section, we reused the example from to show that GAN trained using the MMD loss in Eq. S1 is locally stable. Consider a two-parameter MMD-GAN with uniform latent distribution P Z over [−1, 1], generator G(z) = w 1 z, discriminator D(x) = w 2 x 2, and Gaussian kernel k (a) the data distribution P X is the same as P Z, i.e., uniform over [−1, 1], thus P X is realizable; DISPLAYFORM0 Figure S1: Streamline plots of MMD-GAN using the MMD-rbf and the MMD-rep model on distributions: P Z = U(−1, 1), P X = U(−1, 1) or P X = N. In (a) and (b), the equilibria satisfying P G = P X lie on the line w 1 = 1. In (c), the equilibrium lies around point (1.55, 0.74); in (d), it is around (1.55, 0.32).(b) P X is standard Gaussian, thus non-realizable for any w 1 ∈ R. FIG1 shows that MMD-GAN are locally stable in both cases and D θ * D (x) = 0 does not hold in general for MMD loss. However, MMD-rep may not be globally stable for the tested cases: initialization of (w 1, w 2) in some regions may lead to the trivial solution w 2 = 0 (see FIG1 and S1d). We note that by decreasing the learning rate for G, the area of such regions decreased. At last, it is interesting to note that both MMD-rbf and MMD-rep had the same nontrivial solution w 1 ≈ 1.55 for generator in the non-realizable cases (see FIG1 and S1d). This section divides the proof for Proposition 1 into two parts. First, we show that GAN with the MMD loss in Eq. S1 has equilibria for any parameter configuration of discriminator D; second, we prove the model is locally exponentially stable. For convenience, we consider the general form of discriminator loss in Eq. 5: DISPLAYFORM0 which has L att D and L rep D as the special cases when λ equals −1 and 1 respectively. Consider real data X r ∼ P X, latent variable Z ∼ P Z and generated variable Y g = G θ G (Z). Let x r, z, y g be their samples. Denote ∇. DISPLAYFORM1 where L D and L G are the losses for D and G respectively. Assume an isotropic stationary kernel k(a, b) = k I (a − b) BID6 ) is used in MMD. We first show:Proposition 1 (Part 1). If there exists θ * G ∈ Θ G such that P θ * G = P X, the GAN with the MMD loss in Eq. S1a and Eq. S2 has equilibria (θ * DISPLAYFORM2 where k is the kernel of MMD. The gradients of MMD loss are DISPLAYFORM3 ∼ P G, an unbiased estimator of the squared MMD is BID8) We proceed to prove the model stability. First, following Theorem 5 in BID8 and Theorem 4 in Li et al. (2017a), it is straightforward to see: FORMULA13 ). Consider a non-linear system of parameters (θ, γ): θ = h 1 (θ, γ),γ = h 2 (θ, γ) with an equilibrium point at. Let there exist such that ∀γ ∈ DISPLAYFORM4 DISPLAYFORM5 DISPLAYFORM6 is a Hurwitz matrix, the non-linear system is exponentially stable. Proposition 1 (Part 2). At equilibrium P θ * G = P X, the GAN trained using MMD loss and gradient descent methods is locally exponentially stable at (θ DISPLAYFORM0 ∂b∂c . Based on Eq. S3, we have DISPLAYFORM1 where ⊗ is the kronecker product. At equilibrium, consider a sequence of N samples DISPLAYFORM2 DISPLAYFORM3 Given Lemma A.1 and fact that J GG is the Hessian matrix of M DISPLAYFORM4 is locally constant along some directions in the parameter space of G. As a , null(J GG) ⊆ null(J DG) because varying θ * G along these directions has no effect on D. Following Lemma C.3 of , we consider eigenvalue decomposition DISPLAYFORM5. Thus, the projections γ G = T G θ G are orthogonal to null(J GG). Then, the Jacobian corresponding to the projected system has the form DISPLAYFORM6, where J GG is negative definite. Moreover, on all directions exclude those described by J GG, the system is surrounded by a neighborhood of equilibia at least locally. According to Lemma A.2, the system is exponentially stable. This section shows that constant discriminator output DISPLAYFORM0 may have no discrimination power. First, we make the following assumptions:Assumption 3. 1. D is a multilayer perceptron where each layer l can be factorized into an affine transform and an element-wise activation function f l. 2. Each activation function f l ∈ C 0; furthermore, f l has a finite number of discontinuities and f l ∈ C 06. 3. Input data to D is continuous and its support S is compact in R d with non-zero measure in each dimension and d > 1 7.Based on Assumption 3, we have the following proposition:Proposition 2. If ∀x ∈ S, D(x) = c, where c is constant, then there always exists distortion δx such that x + δx ∈ S and D(x + δx) = c. are model weights and biases, f is an activation function satisfying Assumption 3. For x ∈ S, since D(x) = c, we have h(x) ∈ null(W 2). Furthermore: DISPLAYFORM0 has unique solution for any k ∈ R as long as k · h(x) is within the output range of f. DISPLAYFORM1 and n is the nullity of W 2. Let the projected support beŜ. Thus, DISPLAYFORM2 T with z c = 0.Consider the Jacobian: DISPLAYFORM3 where DISPLAYFORM4 is the input to activation, or pre-activations. SinceŜ is continuous and compact, it has infinite number of boundary points {x b} for d > 1. Consider one boundary pointx b and its normal line δx b. Let > 0 be a small scalar such thatx b − δx b ∈ S andx b + δx b ∈Ŝ.• For linear activation, ∇Σ = I and J is constant. Then z c remains 0 DISPLAYFORM5 there exists z such that h(x + δx) ∈ null(W 2).• For nonlinear activations, assume f has N discontinuities. Since U x T 0 T T + b 1 = c has unique solution for any vector c, the boundary points {x b} cannot yield pre-activations {a b} that all lie on the discontinuities in any of the d h directions. Though we might need to sample d N +1 h points in the worst case to find an exception, there are infinite number of exceptions. Letx b be a sample where {a b} does not lie on the discontinuities in any direction. Because f is continuous, z c remains 0 forx b + δx b, i.e., there exists z such that h(x + δx) ∈ null(W 2).In , we can always find δx such that x + δx / ∈ S and D(x + δx) = c. cannot discriminate against fake samples with distortions to the original data. In contrast, Assumption 2 and Lemma A.1 guarantee that, at equilibrium, the discriminator trained using MMD loss function is effective against such fake samples given a large number of i.i.d. test samples BID8 ). Several loss functions have been proposed to quantify the difference between real and generated sample scores, including: (assume linear activation is used at the last layer of D)• The Minimax loss BID7 ): Softplus(D(G(z) ))] and L G = −L D, which can be derived from the Jensen-Shannon (JS) divergence between P X and the model distribution P G. DISPLAYFORM0 • The non-saturating loss BID7 ), which is a variant of the minimax loss with the same L D and DISPLAYFORM1 • The Hinge loss : DISPLAYFORM2, which is notably known for usage in support vector machines and is related to the total variation (TV) distance .• The Wasserstein loss; BID10 ), which is derived from the Wasserstein distance between P X and P G: DISPLAYFORM3, where D is subject to some Lipschitz constraint.• The maximum mean discrepancy (MMD) (Li et al. (2017a); ), as described in Section 2. For unsupervised image generation tasks on CIFAR-10 and STL-10 datasets, the DCGAN architecture from was used. For CelebA and LSUN bedroom datasets, we added more layers to the generator and discriminator accordingly. See TAB0 and S2 for details. TAB0: DCGAN models for image generation on CIFAR-10 (h = w = 4, H = W = 32) and STL-10 (h = w = 6, H = W = 48) datasets. For non-saturating loss and hinge loss, s = 1; for MMD-rand, MMD-rbf, MMD-rq, s = 16. For a weight matrix W, the spectral norm is defined as σ(W) = max v 2 ≤1 W v 2. The PIM is used to estimate σ(W) , which iterates between two steps: DISPLAYFORM0 The convolutional kernel W c is a tensor of shape h × w × c in × c out with h, w the receptive field size and c in, c out the number of input/output channels. To estimate σ(W c), reshaped it into a matrix W rs of shape (hwc in) × c out and estimated σ(W rs).We propose a simple method to calculate W c directly based on the fact that convolution operation is linear. For any linear map T: R m → R n, there exists matrix W L ∈ R n×m such that y = T (x) can be represented as y = W L x. Thus, we may simply substitute W L = ∂y ∂x in the PIM method to estimate the spectral norm of any linear operation. In the case of convolution operation *, there exist doubly block circulant matrix DISPLAYFORM1 T u which is essentially the transpose convolution of W c on u BID4 ). Thus, similar to PIM, PICO iterates between the following two steps: DISPLAYFORM2 2. Do transpose convolution of W c on u to getv; update v =v/ v 2.Similar approaches have been proposed in and from different angles, which we were not aware during this study. In addition, proposes to compute the exact singular values of convolution kernels using FFT and SVD. In spectral normalization, only the first singular value is concerned, making the power iteration methods PIM and PICO more efficient than FFT and thus preferred in our study. However, we believe the exact method FFT+SVD may eventually inspire more rigorous regularization methods for GAN.The proposed PICO method estimates the real spectral norm of a convolution kernel at each layer, thus enforces an upper bound on the Lipschitz constant of the discriminator D. Denote the upper bound as LIP PICO. In this study, Leaky ReLU (LReLU) was used at each layer of D, thus LIP PICO ≈ 1 . In practice, however, PICO would often cause the norm of the signal passing through D to decrease to zero, because at each layer,• the signal hardly coincides with the first singular-vector of the convolution kernel; and• the activation function LReLU often reduces the norm of the signal. Consequently, the discriminator outputs tend to be similar for all the inputs. To compensate the loss of norm at each layer, the signal is multiplied by a constant C after each spectral normalization. This essentially enlarges LIP PICO by C K where K is the number of layers in the DCGAN discriminator. For all experiments in Section 5, we fixed C = 1 0.55 ≈ 1.82 as all loss functions performed relatively well empirically. In Appendix Section C.3, we tested the effects of coefficient C K on the performance of several loss functions. PIM also enforces an upper bound LIP PIM on the Lipschitz constant of the discriminator D. Consider a convolution kernel W c with receptive field size h × w and stride s. Let σ PICO and σ PIM be the spectral norm estimated by PICO and PIM respectively. We empirically In this section, we empirically evaluate the effects of coefficient C K on the performance of PICO and compare PICO against PIM using several loss functions. We used a similar setup as Section 5.1 with the following adjustments. Four loss functions were tested: hinge, MMD-rbf, MMD-rep and MMD-rep-b. Either PICO or PIM was used at each layer of the discriminator. For PICO, five coefficients C K were tested: 16, 32, 64, 128 and 256 (note this is the overall coefficient for K layers; K = 8 for CIFAR-10 and STL-10; K = 10 for CelebA and LSUN-bedroom; see Appendix B.2). FID was used to evaluate the performance of each combination of loss function and power iteration method, e.g., hinge + PICO with C K = 16.Results: For each combination of loss function and power iteration method, the distribution of FID scores over 16 learning rate combinations is shown in FIG0. We separated well-performed learning rate combinations from diverged or poorly-performed ones using a threshold τ as the diverged cases often had non-meaningful FID scores. The boxplot shows the distribution of FID scores for goodperformed cases while the number of diverged or poorly-performed cases was shown above each box if it is non-zero. 1) When PICO was used, the hinge, MMD-rbf and MMD-rep methods were sensitive to the choices of C K while MMD-rep-b was robust. For hinge and MMD-rbf, higher C K may in better FID scores and less diverged cases over 16 learning rate combinations. For MMD-rep, higher C K may cause more diverged cases; however, the best FID scores were often achieved with C K = 64 or 128.2) For CIFAR-10, STL-10 and CelebA datasets, PIM performed comparable to PICO with C K = 128 or 256 on four loss functions. For LSUN bedroom dataset, it is likely that the performance of PIM corresponded to that of PICO with C K > 256. This implies that PIM may in a relatively loose Lipschitz constraint on deep convolutional networks.3) MMD-rep-b performed generally better than hinge and MMD-rbf with tested power iteration methods and hyper-parameter configurations. Using PICO, MMD-rep also achieved generally better FID scores than hinge and MMD-rbf. This implies that, given a limited computational budget, the proposed repulsive loss may be a better choice than the hinge and MMD loss for the discriminator. TAB2 shows the best FID scores obtained by PICO and PIM where C K was fixed at 128 for hinge and MMD-rbf, and 64 for MMD-rep and MMD-rep-b. For hinge and MMD-rbf, PICO performed significantly better than PIM on the LSUN-bedroom dataset and comparably on the rest datasets. For MMD-rep and MMD-rep-b, PICO achieved consistently better FID scores than PIM.However, compared to PIM, PICO has a higher computational cost which roughly equals the additional cost incurred by increasing the batch size by two . This may be problematic when a small batch has to be used due to memory constraints, e.g., when handling high resolution images on a single GPU. Thus, we recommend using PICO when the computational cost is less of a concern. D SUPPLEMENTARY EXPERIMENTS D.1 LIPSCHITZ CONSTRAINT VIA GRADIENT PENALTY Gradient penalty has been widely used to impose the Lipschitz constraint on the discriminator arguably since Wasserstein GAN BID10 ). This section explores whether the proposed repulsive loss can be applied with gradient penalty. Several gradient penalty methods have been proposed for MMD-GAN. penalized the gradient norm of witness function y) ] w.r.t. the interpolated sample z = ux + (1 − u)y to one, where u ∼ U 9. More recently, proposed to impose the Lipschitz constraint on the mapping φ • D directly and derived the Scaled MMD (SMMD) as SM k (P, Q) = σ µ,k,λ M k (P, Q), where the scale σ µ,k,λ incorporates gradient and smooth penalties. Using the Gaussian kernel and measure µ = P X leads to the discriminator loss: DISPLAYFORM0 DISPLAYFORM1 We apply the same formation of gradient penalty to the repulsive loss: DISPLAYFORM2 where the numerator L rep D − 1 ≤ 0 so that the discriminator will always attempt to minimize both L rep D and the Frobenius norm of gradients ∇D(x) w.r.t. real samples. Meanwhile, the generator is trained using the MMD loss L mmd G (Eq. 2).Experiment setup: The gradient-penalized repulsive loss L rep-gp D (Eq. S8, referred to as MMD-repgp) was evaluated on the CIFAR-10 dataset. We found λ = 10 in too restrictive 9 Empirically, we found this gradient penalty did not work with the repulsive loss. The reason may be the attractive loss L att D (Eq. 3) is symmetric in the sense that swapping P X and PG in the same loss; while the repulsive loss is asymmetric and naturally in varying gradient norms in data space. and used λ = 0.1 instead. Same as, the output dimension of discriminator was set to one. Since we entrusted the Lipschitz constraint to the gradient penalty, spectral normalization was not used. The rest experiment setup can be found in Section 5.1. TAB3 shows that the proposed repulsive loss can be used with gradient penalty to achieve reasonable on CIFAR-10 dataset. For comparison, we cited the Inception score and FID for Scaled MMD-GAN (SMMDGAN) and Scaled MMD-GAN with spectral normalization (SN-SMMDGAN) from. Note that SMMDGAN and SN-SMMDGAN used the same DCGAN architecture as MMD-rep-gp, but were trained for 150k generator updates and 750k discriminator updates, much more than that of MMD-rep-gp (100k for both G and D). Thus, the repulsive loss significantly improved over the attractive MMD loss for discriminator. In this section, we investigate the impact of the output dimension of discriminator on the performance of repulsive loss. Experiment setup: We used a similar setup as Section 5.1 with the following adjustments. The repulsive loss was tested on the CIFAR-10 dataset with a variety of discriminator output dimensions: d ∈ {1, 4, 16, 64, 256}. Spectral normalization was applied to discriminator with the proposed PICO method (see Appendix C) and the coefficients C K selected from {16, 32, 64, 128, 256}.Results: TAB4 shows that using more than one output neuron in the discriminator D significantly improved the performance of repulsive loss over the one-neuron case on CIFAR-10 dataset. The reason may be that using insufficient output neurons makes it harder for the discriminator to learn an injective and discriminative representation of the data (see FIG8). However, the performance gain diminished when more neurons were used, perhaps because it becomes easier for D to surpass the generator G and trap it around saddle solutions. The computation cost also slightly increased due to more output neurons. Generated samples on CelebA dataset are given in FIG7 and LSUN bedrooms in FIG8. Spectral normalization was applied to discriminator with two power iteration methods: PICO and PIM. For PICO, five coefficients C K were tested: 16, 32, 64, 128, and 256. A learning rate combination was considered diverged or poorly-performed if the FID score exceeded a threshold τ, which is 50, 80, 50, 90 for CIFAR-10, STL-10, CelebA and LSUN-bedroom respectively. The box quartiles were plotted based on the cases with FID < τ while the number of diverged or poorly-performed cases (out of 16 learning rate combinations) was shown above each box if it is non-zero. We introduced τ because the diverged cases often had arbitrarily large and non-meaningful FID scores. DISPLAYFORM0

Rearranging the terms in maximum mean discrepancy yields a much better loss function for the discriminator of generative adversarial nets

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.