Question: Represent the text to find a very short summary (1-2 sentences max)

Question: A fundamental, and still largely unanswered, question in the context of Generative Adversarial Networks (GANs) is whether GANs are actually able to capture the key characteristics of the datasets they are trained on. The current approaches to examining this issue require significant human supervision, such as visual inspection of sampled images, and often offer only fairly limited scalability. In this paper, we propose new techniques that employ classification-based perspective to evaluate synthetic GAN distributions and their capability to accurately reflect the essential properties of the training data. These techniques require only minimal human supervision and can easily be scaled and adapted to evaluate a variety of state-of-the-art GANs on large, popular datasets. They also indicate that GANs have significant problems in reproducing the more distributional properties of the training dataset. In particular, the diversity of such synthetic data is orders of magnitude smaller than that of the original data. Generative Adversarial Networks (GANs) BID6 have garnered a significant amount of attention due to their ability to learn generative models of multiple natural image datasets BID11 BID3 BID7 ). Since their conception, a fundamental question regarding GANs is to what extent they truly learn the underlying data distribution. This is a key issue for multiple reasons. From a scientific perspective, understanding the capabilities of common GANs can shed light on what precisely the adversarial training setup allows the GAN to learn. From an engineering standpoint, it is important to grasp the power and limitations of the GAN framework when applying it in concrete applications. Due to the broad potential applicability of GANs, researchers have investigated this question in a variety of ways. When we evaluate the quality of a GAN, an obvious first check is to establish that the generated samples lie in the support of the true distribution. In the case of images, this corresponds to checking if the generated samples look realistic. Indeed, visual inspection of generated images is currently the most common way of assessing the quality of a given GAN. Individual humans can performs this task quickly and reliably, and various GANs have achieved impressive for generating realistic-looking images of faces and indoor scenes BID13 BID3.Once we have established that GANs produce realistic-looking images, the next concern is that the GAN might simply be memorizing the training dataset. While this hypothesis cannot be ruled out entirely, there is evidence that GANs perform at least some non-trivial modeling of the unknown distribution. Previous studies show that interpolations in the latent space of the generator produce novel and meaningful image variations BID11, and that there is a clear disparity between generated samples and their nearest neighbors in the true dataset BID1.Taken together, these provide evidence that GANs could constitute successful distribution learning algorithms, which motivates studying their distributions in more detail. The direct approach is to compare the probability density assigned by the generator with estimates of the true distribution BID16. However, in the context of GANs and high-dimensional image distributions, this is complicated by two factors. First, GANs do not naturally provide probability estimates for their samples. Second, estimating the probability density of the true distribution is a challenging problem itself (the adversarial training framework specifically avoids this issue). Hence prior work has only investigated the probability density of GANs on simple datasets such as MNIST BID16.Since reliably computing probability densities in high dimensions is challenging, we can instead study the behavior of GANs in low-dimensional problems such as two-dimensional Gaussian mixtures. Here, a common failure of GANs is mode collapse, wherein the generator assigns a disproportionately large mass to a subset of modes from the true distribution BID5. This raises concerns about a lack of diversity in the synthetic GAN distributions, and recent work shows that the learned distributions of two common GANs indeed have (moderately) low support size for the CelebA dataset BID1. However, the approach of BID1 heavily relies on a human annotator in order to identify duplicates. Hence it does not easily scale to comparing many variants of GANs or asking more fine-grained questions than collision statistics. Overall, our understanding of synthetic GAN distributions remains blurry, largely due to the lack of versatile tools for a quantitative evaluation of GANs in realistic settings. The focus of this work is precisly to address this question:Can we develop principled and quantitative approaches to study synthetic GAN distributions?To this end, we propose two new evaluation techniques for synthetic GAN distributions. Our methods are inspired by the idea of comparing moments of distributions, which is at the heart of many methods in classical statistics. Although simple moments of high-dimensional distributions are often not semantically meaningful, we can extend this idea to distributions of realistic images by leveraging image statistics identified using convolutional neural networks. In particular, we train image classifiers in order to construct test functions corresponding to semantically meaningful properties of the distributions. An important feature of our approach is that it requires only light human supervision and can easily be scaled to evaluating many GANs and large synthetic datasets. Using our new evaluation techniques, we study five state-of-the-art GANs on the CelebA and LSUN datasets, arguably the two most common testbeds for advanced GANs. We find that most of the GANs significantly distort the relative frequency of even basic image attributes, such as the hair style of a person or the type of room in an indoor scene. This clearly indicates a mismatch between the true and synthetic distributions. Moreover, we conduct experiments to explore the diversity of GAN distributions. We use synthetic GAN data to train image classifiers and find that these have significantly lower accuracy than classifiers trained on the true data set. This points towards a lack of diversity in the GAN data, and again towards a discrepancy between the true and synthetic distributions. In fact, our additional examinations show that the diversity in GANs is only comparable to a subset of the true data that is 100× smaller. When comparing two distributions, a common first test is to compute low-order moments such as the mean and the variance. If the distributions are simple enough, these quantities provide a good understanding for how similar they are. Moreover, low-order moments have a precise definition and are usually quick to compute. On the other hand, low-order moments can also be misleading for more complicated, high-dimensional distributions. As a concrete example, consider a generative model of digits (such as MNIST). If a generator produces digits that are shifted by a significant amount yet otherwise perfect, we will probably still consider this as a good approximation of the true distribution. However, the expectation (mean moment) of the generator distribution can be very different from the expectation of the true data distribution. This raises the question of what other properties of high-dimensional image distributions are easy to test yet semantically meaningful. In the next two subsections, we describe two concrete approaches to evaluate synthetic GAN data that are easy to compute yet capture relevant information about the distribution. The common theme is that we employ convolutional neural networks in order to capture properties of the distributions that are hard to describe in a mathematically precise way, but usually well-defined for a human (e.g., what fraction of the images shows a smiling person?). Automating the process of annotating images with such high-level information will allow us to study various aspects of synthetic GAN data. Mode collapse refers to the tendency of the generator to concentrate a large probability mass on a few modes of the true distribution. While there is ample evidence for the presence of mode-collapse in GANs BID5 BID1 BID10, elegant visualizations of this phenomena are somewhat restricted to toy problems on low-dimensional distributions BID5 BID10. For image datasets, it is common to rely on human annotators and derived heuristics (see Section 2.3). While these methods have their merits, they are restrictive both in the scale and granularity of testing. Here we propose a classification-based tool to assess how good GANs are at assigning the right mass across broad concepts/modes. To do so, we use a trained classifier as an expert "annotator" that labels important features in synthetic data, and then analyze the ing distribution. Specifically, our goal is to investigate if a GAN trained on a well-balanced dataset (i.e., contains equal number of samples from each class) can learn to reproduce this balanced structure. Let DISPLAYFORM0 represent a dataset of size N with C classes, where (x i, y i) denote an image-label pair drawn from true data. If the dataset D is balanced, it contains N/C images per class. The procedure for computing class distribution in synthetic data is:1. Train an annotator (a multi-class classifier) using the dataset D. 2. Train an unconditional GAN on the images X from dataset D, without using class labels. 3. Create a synthetic dataset by sampling N images from a GAN and labeling them using the annotator from Step 1.The annotated data generated via the above procedure can provide insight into the GAN's class distribution at the scale of the entire dataset. Moreover, we can vary the granularity of mode analysis by choosing richer classification tasks, i.e., more challenging classes or a larger number of them. In Section 3.3, we use this technique to visualize mode collapse in several state-of-the-art GANs on the CelebA and LSUN datasets. All the studied GANs show significant mode collapse and the effect becomes more pronounced when the granularity of the annotator is increased (larger number of classes). We also investigate the temporal aspect of the GAN setup and find that the dominant mode varies widely over the course of training. Our approach also enables us to benchmark and compare GANs on different datasets based on the extent of mode collapse in the learned distributions. Our above method for inspecting distribution of modes in synthetic data provides a coarse look at the statistics of the underlying distribution. While the ing quantities are semantically meaningful, they capture only simple notions of diversity. To get a more holistic view on the sample diversity in the synthetic distribution, we now describe a second classification-based approach for evaluating GAN distributions. The main question that motivates it is: Can GANs recover the key aspects of real data to enable training a good classifier? We believe that this is an interesting measure of sample diversity for two reasons. First, classification of high-dimensional image data is a challenging problem, so a good training dataset will require a sufficiently diverse sample from the distribution. Second, augmenting data for classification problems is one of the proposed use cases of GANs (e.g., see the recent work of BID14).If GANs are truly able to capture the quality and diversity of the underlying data distribution, we expect almost no gap between classifiers trained on true data and synthetic data from a GAN. A generic method to produce data from GANs for classification is to train separate GANs for each class in the dataset D. 1 Samples from these class-wise GANs can then be pooled together to get a labeled synthetic dataset. Note that the labels are trivially determined based on the class modeled by the particular GAN from which a sample is drawn. We perform the following steps to assess the classification performance of synthetic data vs. true data:1. Train a classifier on the true data D (from Section 2.1) as a benchmark for comparison. 2. Train C separate unconditional GANs, one per class in dataset D. 3. Generate a balanced synthetic labeled dataset of size N by consolidating an equal number of samples drawn from each of these C GANs. The labels obtained by aggregating samples from per-class GANs are designated as "default" labels for the synthetic dataset. Note that by design, both true and synthetic datasets have N samples, with N/C examples per class. 4. Use synthetic labeled data from Step 3 to train classifier with the same architecture as Step 1.Comparing the classifiers from Steps 1 and 4 can then shed light on the disparity between the two distributions. BID11 conducted an experiment similar to Step 2 on the MNIST dataset using a conditional GAN. They found that samples from their DCGAN performed comparably to true data on nearest neighbor classification. We obtained similar good on MNIST, which could be due to the efficacy of GANs in learning the MNIST distribution or due to the ease of getting good accuracy on MNIST even with a small training set BID12. To clarify this question, we restrict our analysis to more complex datasets, specifically CelebA and LSUN.We evaluate the two following properties in our classification task: (i) How well can the GANs recover nuances of the decision boundary, which is reflected by how easily the classifier can fit the training data?(ii) How does the diversity of synthetic data compare to that of true data when measured by classification accuracy on a hold-out set of true data?We observe that all the studied GANs have very low diversity in this metric. In particular, the accuracy achieved by a classifier trained on GAN data is comparable only to the accuracy of a classifier trained on a 100× (or more) subsampled version of the true dataset. Even if we draw more samples from the GANs to produce a training set several times larger than the true dataset, there is no improvement in performance. Looking at the classification accuracy gives us a way to compare different models on a potential downstream application of GANs. Interestingly, we find that visual quality of samples does not necessarily correlate with good classification performance. In GAN literature, it is common to investigate performance using metrics that involve human supervision. BID1 proposed a measure based on manually counting duplicates in GAN samples as a heuristic for the support or diversity of the learned distribution. In BID16, manual classification of a small sample (100 images) of GAN generated MNIST images is used as a test for the GAN is missing certain modes. Such annotator-based metrics have clear advantages in identifying relevant failure-modes of synthetic samples, which explains why visual inspection (eyeballing) is still the most popular approach to assess GAN samples. There have also been various attempts to build good metrics for GANs that are not based on manual heuristics. Parzen window estimation can be used to approximate the log-likelihood of the distribution, though it is known to work poorly for high-dimensional data BID15. BID16 develop a method to get a better estimate for log-likelihood using annealed importance sampling. BID13 propose a metric known as Inception Score, where the entropy in the labels predicted by a pre-trained Inception network is used to assess the diversity in GAN samples. In the following sub-sections we describe the setup and for our classification-based GAN benchmarks. Additional details can be found in Section 5 in the Appendix. GANs have shown promise in generating realistic samples, ing in efforts to apply them to a broad spectrum of datasets. However, the Large-scale CelebFaces Attributes (CelebA) BID9 and Large-Scale Scene Understanding (LSUN) BID17 datasets remain the most popular and canonical ones in developing and evaluating GAN variants. Conveniently, these datasets also have rich annotations, making them particularly suited for our classification-based evaluations. Details on the setup for classification tasks for these datasets are given in the Appendix (Section 5). Using our framework, we perform a comparative study of several popular variants of GANs: 1. Deep Convolutional GAN (DCGAN): Convolutional GAN trained using a Jensen-Shannon divergence-based objective BID6 BID11. 2. Wasserstein GAN (WGAN): GAN that uses a Wasserstein distance-based objective. 3. Adversarially Learned Inference (ALI): GAN that uses joint adversarial training of generative and inference networks BID4. 4. Boundary Equilibrium GAN (BEGAN): Auto-encoder style GAN trained using Wasserstein distance objective BID2. Figure 1: Visualizations of mode collapse in the synthetic, GAN-generated data produced after training on our chosen subsets of CelebA and LSUN datasets. Left panel shows the relative distribution of classes in samples drawn from synthetic datasets extracted at the end of the training process, and compares is to the true data distribution (leftmost plots). On the right, shown is the evolution of analogous class distribution for different GANs over the course of training. BEGAN did not converge on the LSUN tasks and hence is excluded from the corresponding analysis.5. Improved GAN (ImGAN): GAN that uses semi-supervised learning (labels are part of GAN training), with various other architectural and procedural improvements BID13. All the aforementioned GANs are unconditional, however, ImGAN has access to class labels as a part of the semi-supervised training process. We use standard implementations for each of these models, details of which are provided in the Appendix (Section 5). We also used the prescribed hyper-parameter settings for each GAN, including number of iterations we train them for. Our analysis is based on 64 × 64 samples, which is a size at which GAN generated samples tend to be of high quality. We also use visual inspection to ascertain that the perceptual quality of GAN samples in our experiments is comparable to those reported in previous studies. We demonstrate sample images in FIG1 in the Appendix. BEGAN did not converge in our experiments on the LSUN dataset and hence is excluded from the corresponding analysis. In our study, we use two types of classification models: 1. ResNet: 32-Layer Residual network BID7. Here, we choose a ResNet as it is a standard classifier in vision and yields high accuracy on various datasets, making it a reliable baseline. 2. Linear Model: This is a network with one-fully connected layer between the input and output (no hidden layers) with a softmax non-linearity. If the dimensions of input x and outputŷ, are D and C (number of classes) respectively, then linear models implement the functionŷ = σ(DISPLAYFORM0 where W is a D × C matrix, b is a C × 1 vector and σ(·) is the softmax function. Due to it's simplicity, this model will serve as a useful baseline in some of our experiments. We always train the classifiers to convergence, with decaying learning rate and no data augmentation. Experimental for quantifying mode collapse through classification tasks, described in Section 2.1, are presented below. Table 2 in the Appendix gives details on datasets (subsets of CelebA and LSUN) used in our analysis, such as size (N), number of classes (C), and accuracy of the annotator, i.e., a classifier pre-trained on true data, which is then used to label the synthetic, GANgenerated data. Figure 1 presents class distribution in synthetic data, as specified by these annotators. The left panel compares the relative distribution of modes in true data (uniform) with that in various GAN-generated datasets. Each of these datasets is created by drawing N samples from the GAN after it was trained on the corresponding true dataset. The right panel illustrates the evolution of class distributions in various GANs over the course of training 2.Results: These visualization lead to the following findings:• All GANs seem to suffer from significant mode-collapse. This becomes more apparent when the annotator granularity is increased, by considering a larger set of classes. For instance, one should compare the relatively balanced class distributions in the 3-class LSUN task to the near-absence of some modes in the 5-class task.• Mode collapse is prevalent in GANs throughout the training process, and does not seem to recede over time. Instead the dominant mode(s) often fluctuate wildly over the course of the training.• For each task, often there is a common set of modes onto which the distint GANs exhibit collapse. In addition to viewing our method as an approach to analyze the mode collapse, we can also use it as a benchmark for GAN comparison. From this perspective, we can observe that, on CelebA, DCGAN and ALI learn somewhat balanced distributions, while WGAN, BEGAN and Improved GAN show prominent mode collapse. This is in contrast to the obtained LSUN, where, for example, WGAN exhibit relatively small mode collapse, while ALI suffers from significant mode collapse even on the simple 3-class task. This highlights the general challenge in real world applications of GANs: they often perform well on the datasets they were designed for (e.g. ALI on CelebA and WGAN on LSUN), but extension to new datasets is not straightforward. Temporal analysis of mode-collapse shows that there is wide variation in the dominant mode for WGAN and Improved GAN, whereas for BEGAN, the same mode(s) often dominates the entire training process. Using the procedure outlined in Section 2.2, we perform a quantitative assessment of sample diversity in GANs on the CelebA and LSUN datasets. We restrict our experiments to binary classification as we find they have sufficient complexity to highlight the disparity between true and synthetic data. Selected for classification-based evaluation of GANs are presented in Table 1. A more extensive study is presented in Table 3, and Figures 4 and 5 in the Appendix (Section 5).As a preliminary check, we inspect the quality of our labeled GAN datasets. For this, we use high-accuracy annotators from Section 2.1 to predict labels for GAN generated data and measure consistency between the predicted and default labels (label correctness). We also inspect confidence scores, defined as the softmax probabilities for predicted class, of the annotator. The motivation behind these metrics is that if the classifier can correctly and with high-confidence predict labels for labeled GAN samples, then it is likely that they are convincing examples of that class, and hence of good "quality". Empirical for label agreement and annotator confidence of GAN generated datasets are shown in Tables 1 and 3, and FIG4. We also report an equivalent Inception Score BID13 ), similar to that described in Section 2.3. Using the Inception network to get the label distribution may not be meaningful for face or scene images. Instead, we compute the Inception Score using the label distribution predicted from the annotator networks. Score is computed as exp(E x [KL(p(y|x))||p(y)]), where y refers to label predictions from the annotators 3. Table 1: Select from the comparative study on classification performance of true data vs. GANs on the CelebA and LSUN datasets. Label correctness measures the agreement between default labels for the synthetic datasets, and those predicted by the annotator, a classifier trained on true data. Shown alongside are the equivalent inception scores computed using labels predicted by the annotator (rather than an Inception Network). Training and test accuracies for a linear model on the various true and synthetic datasets are reported. Also presented are the corresponding accuracies for this classifier trained on down-sampled true data (↓ M) and oversampled synthetic data (↑ L). Test accuracy for ResNets trained on these datasets is also shown (training accuracy was always 100%), though it is noticeable that deep networks suffer from issues when trained on synthetic datasets. Next, we train classifiers using the true and labeled GAN-generated datasets and study their performance in terms of accuracy on a hold-out set of true data. ResNets (and other deep variants) yield good classification performance on true data, but suffer from severe overfitting on the synthetic data, leading to poor test accuracy. This already indicates a possible problem with GANs and the diversity of the data they generate. But to highlight this problem better and avoid the issues that stem from overfitting, we also look for a classifier which does not always overfit on the synthetic data. We, however, observed that even training simple networks, such as one fully connected layer with few hidden units, led to overfitting on synthetic data. Hence, we resorted to a very basic linear model described in Section 3.2. Tables 1 and 3 shows from binary classification experiments using linear models, with the training and test accuracies of the classifier on various datasets. Finally, to get a better understanding of the underlying "diversity" of synthetic datasets, we train linear models using down-sampled versions of true data (no augmentation), and compare this to the performance of synthetic data, as shown in Tables 1 and 3. Down-sampling the data by a factor of M, denoted as ↓ M implies selecting a random N/M subset of the data D. Visualizations of how GAN classification performance compares with (down-sampled) true data are in Figure 5 in the Appendix. A natural argument in the defense of GANs is that we can oversample them, i.e. generate datasets much larger than the size of training data. Results for linear models trained using a 10-fold oversampling of GANs (drawing 10N samples), denoted by ↑ 10, are show in Tables 1 and 3.Results: The major findings from these experiments are:• Based on Tables 1 and 3, and FIG4, we see strong agreement between annotator labels and true labels for synthetic data, on par with the scores for the test set of true data. It is thus apparent that the GAN images are of high-quality, as expected based on the visual inspection. These scores are lower for LSUN than CelebA, potentially due to lower quality of generated LSUN images. From these , we can get a broad understanding of how good GANs are at producing convincing/representative samples from different classes across datasets. This also shows that simple classification-based benchmarks can highlight relevant properties of synthetic datasets.• The equivalent inception score is not very informative and is similar for the true (hold-out set) and synthetic datasets. This is not surprising given the simple nature of our binary classification task and the fact that the true and synthetic datasets have almost a uniform distribution over labels.• It is evident from Table 1 that there is a large performance gap between true and synthetic data on classification tasks. Inspection of training accuracies shows that linear models are able to nearly fit the synthetic datasets, but are grossly underfitting on true data. Given the high scores of synthetic data on the previous experiments to assess dataset'quality' (Tables 1 and 3, and FIG4, it is likely that the poor classification performance is more indicative of lack of 'diversity'.• Comparing GAN performance to that of down-sampled true data reveals that the learned distribution, which was trained on datasets that have around hundred thousand data points exhibits diversity that is on par with what only mere couple of hundreds of true data samples constitute! This shows that, at least from the point of view of classification, the diversity of the GAN generated data is severely lacking.• Oversampling GANs by 10-fold to produce larger datasets does not improve classification performance. The disparity between true and synthetic data remains nearly unchanged even after this significant oversampling, further highlighting the lack of diversity in GANs. In terms of the of relative performance of various GANs, we observe that WGAN and ALI (on CelebA) perform better than the other GANs. While BEGAN samples have good perceptual quality (see FIG1, they consistently perform badly on our classification tasks. On the other hand, WGAN samples have relatively poor visual quality but seem to outperform other GANs in classification tasks. This is a strong indicator of the need to consider other metrics, such as the ones proposed in this paper, in addition to visual inspection to study GANs. For LSUN, the gap between true and synthetic data is much larger, with the classifiers getting near random performance on all the synthetic datasets. Note that these classifiers get poor test accuracy on LSUN but are not overfitting on the training data. In this case, we speculate the lower performance could be due to both lower quality and diversity of LSUN samples. In summary, our key experimental finding is that even simple classification-based tests can hold tremendous potential to shed insight on the learned distribution in GANs. This not only helps us to get a deeper understanding of many of the underlying issues, but also provides with a more quantitative and rigorous platform on which to compare different GANs. Our techniques could, in principle, be also applied to assess other generative models such as Variational Auto-Encoders (VAEs) BID8. However, VAEs have significant problems in generating realistic samples on the datasets used in our analysis in the first place -see BID1. In this paper, we put forth techniques for examining the ability of GANs to capture key characteristics of the training data, through the lens of classification. Our tools are scalable, quantitative and automatic (no need for visual inspection of images). They thus are capable of studying state-ofthe-art GANs on realistic, large-scale image datasets. Further, they serve as a mean to perform a nuanced comparison of GANs and to identify their relative merits, including properties that cannot be discerned from mere visual inspection. We then use the developed techniques to perform empirical studies on popular GANs on the CelebA and LSUN datasets. Our examination shows that mode collapse is indeed a prevalent issue for GANs. Also, we observe that synthetic GAN-generated datasets have significantly reduced diversity, at least when examined from a classification perspective. In fact, the diversity of such synthetic data is often few orders of magnitude smaller than that of the true data. Furthermore, this gap in diversity does not seem to be bridged by simply producing much larger datasets by oversampling GANs. Finally, we also notice that good perceptual quality of samples does not necessarily correlate -and might sometime even anti-correlate -with distribution diversity. These findings suggest that we need to go beyond the visual inspection-based evaluations and look for more quantitative tools for assessing quality of GANs, such as the ones presented in this paper. To assess GAN performance from the perspective of classification, we construct a set of classification tasks on the CelebA and LSUN datasets. In the case of the LSUN dataset, images are annotated with scene category labels, which makes it straightforward to use this data for binary and multiclass classification. On the other hand, each image in the CelebA dataset is labeled with 40 binary attributes. As a , a single image has multiple associated attribute labels. Here, we construct classification tasks can by considering binary combinations of an attribute(s) (examples are shown in FIG1). Attributes used in our experiments were chosen such that the ing dataset was large, and classifiers trained on true data got high-accuracy so as to be good annotators for the synthetic data. Details on datasets used in our classification tasks, such as training set size (N), number of classes (C), and accuracy of the annotator, i.e., a classifier pre-trained on true data which is used to label the synthetic GAN-generated data, are provided in Table 2. Table 2: Details of CelebA and LSUN subsets used for the studies in Section 3.3. Here, we use a classifier trained on true data as an annotator that let's us infer label distribution for the synthetic, GAN-generated data. N is the size of the training set and C is the number of classes in the true and synthetic datasets. Annotator's accuracy refers to the accuracy of the classifier on a test set of true data. For CelebA, we use a combination of attribute-wise binary classifiers as annotators due their higher accuracy compared to a single classifier trained jointly on all the four classes. Benchmarks were performed on standard implementations - For each of our benchmark experiments, we ascertain that the visual quality of samples produced by the GANs is comparable to that reported in prior work. Examples of random samples drawn for multi-class datasets from both true and synthetic data are shown in FIG1 for the CelebA dataset, and in FIG3 for the LSUN dataset. In the studies to observe mode collapse in GANs described in Sections 2.1 and 3.3, we use a pretrained classifier as an annotator to obtain the class distribution for datasets generated from unconditional GANs. FIG4 shows histograms of annotator confidence for the datasets used for benchmarking listed in Table 2. As can be seen in these figures, the annotator confidence for the synthetic data is comparable to that on the hold-out set of true data. Thus, it seems likely that the GAN generated samples are of good quality and are truly representative examples of their respective classes, as expected based on visual inspection. Table 3 presents an extension of the comparative study of classification performance of true and GAN generated data provided in Table 1. Visualizations of how test accuracies of a linear model classifier trained on GAN data compares with one trained on true data is shown in Figure 5 obtained by drawing N samples from GANs at the culmination of the training process. Based on these visualizations, it is apparent that GANs have comparable classification performance to a subset of training data that is more than a 100x smaller. Thus, from the perspective of classification, GANs have diversity on par with a few hundred true data samples. CelebA: Smiling/Not Smiling CelebA: Black Hair/Not Black Hair LSUN: Bedroom/Kitchen Figure 5: Illustration of the classification performance of true data compared with GAN-generated synthetic datasets based on experiments described in Section 3.4. Classification is performed using a basic linear model, described in Section 3.2, and performance is reported in terms of accuracy on a hold-out set of true data. In the plots, the bold curve captures the classification performance of models trained on true data vs the size of the true dataset (maximum size is N). Dashed lines represent performance of classifiers trained on various GAN-generated datasets of size N. These plots indicate that GAN samples have "diversity" comparable to a small subset (few hundred samples) of true data. Here the notion of diversity is one that is relevant for classification tasks. Table 3: Detailed version of the comparative study of the classification performance of true data and GANs on the CelebA and LSUN datasets shown in Table 1, based on experiments described in Section 3.4. Label correctness measures the agreement between default labels for the synthetic datasets, and those predicted by the annotator, a classifier trained on the true data. Shown alongside are the equivalent inception scores computed using labels predicted by the annotator (instead of the Inception Network). Training and test accuracies for a linear model classifier on the various true and synthetic datasets are reported. Also presented are the corresponding accuracies for a linear model trained on down-sampled true data (↓ M) and oversampled synthetic data (↑ L). Test accuracy for ResNets trained on these datasets is also shown (training accuracy was always 100%), though it is noticeable that deep networks suffer from issues when trained on synthetic datasets.

We propose new methods for evaluating and quantifying the quality of synthetic GAN distributions from the perspective of classification tasks

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We introduce the NLC, a metric that is cheap to compute in the networks randomly initialized state and is highly predictive of generalization, at least in fully-connected networks.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: The goal of survival clustering is to map subjects (e.g., users in a social network, patients in a medical study) to $K$ clusters ranging from low-risk to high-risk. Existing survival methods assume the presence of clear \textit{end-of-life} signals or introduce them artificially using a pre-defined timeout. In this paper, we forego this assumption and introduce a loss function that differentiates between the empirical lifetime distributions of the clusters using a modified Kuiper statistic. We learn a deep neural network by optimizing this loss, that performs a soft clustering of users into survival groups. We apply our method to a social network dataset with over 1M subjects, and show significant improvement in C-index compared to alternatives. Free online subscription services (e.g., Facebook, Pandora) use survival models to predict the relationship between observed subscriber covariates (e.g. usage patterns, session duration, gender, location, etc.) and how long a subscriber remains with an active account BID26 BID11. Using the same tools, healthcare providers make extensive use of survival models to predict the relationship between patient covariates (e.g. smoking, administering drug A or B) and the duration of a disease (e.g., herpes, cancer, etc.). In these scenarios, rarely there is an end-of-life signal: non-paying subscribers do not cancel their accounts, tests rarely declare a patient cancer-free. We want to assign subjects into K clusters, ranging from short-lived to long-lived subscribers (diseases).Despite the recent community interest in survival models BID1 BID33, existing survival analysis approaches require an unmistakable end-of-life signal (e.g., the subscriber deletes his or her account, the patient is declared disease-free), or a pre-defined endof-life "timeout" (e.g., the patient is declared disease-free after 5 years, the subscriber is declared permanently inactive after 100 days of inactivity). Methods that require end-of-life signals also include BID23 BID8 BID3 BID14 BID24 BID29 BID31 BID47 BID9 BID19 BID41 BID40 BID17 BID48 BID26 BID0 BID4 BID5 BID35 BID46 BID30.In this work, we propose to address the lifetime clustering problem without end-of-life signals for the first time, to the best of our knowledge. We begin by describing two possible datasets where such a clustering approach could be applied.• Social Network Dataset: Users join the social network at different times and participate in activities defined by the social network (login, send/receive comments). The covariates are the various attributes of a user like age, gender, number of friends, etc., and the inter-event time is the time between user's two consecutive activities. In this case, censoring is due to a fixed point of data collection that we denote t m, the time of measurement. Thus, time till censoring for a particular user is the time from her last activity to t m. Lifetime of a user is defined as the time from her joining till she permanently deletes her account.• Medical Dataset: Subjects join the medical study at the same time and are checked for the presence of a particular disease. The covariates are the attributes of the disease-causing cell in subject, inter-event time is the time between two consecutive observations of the presence of disease. The time to censoring is the difference between the time of last observation when the disease was present and the time of final observation. If the final observation for a subject indicates presence of the disease, then time to censoring is zero. Lifetime of the disease is defined as the time between the first observation of the disease and the time until it is permanently cured. We use a deep neural network and a new loss function, with a corresponding backpropagation modification, for clustering subjects without end-of-life signals. We are able to overcome the technical challenges of this problem, in part, thanks to the ability of deep neural networks to generalize while overfitting the training data BID49. The task is challenging for the following reasons:• The problem is fully unsupervised, as there is no pre-defined end-of-life timeout. While semisupervised clustering approaches exist BID0 BID4 BID5 BID35 BID46, they assume that end-of-life signals appearing before the observation time are observed; to the best of our knowledge, there are no fully unsupervised approach that can take complex input variables.• There is no hazard function that can be used to define the "cure" rate, as we cannot determine whether the disease is cured, or whether the subscriber will never return to the website, without observing for an infinitely long time.• Cluster assignments may depend on highly complex interactions between the observed covariates and the observed events. The unobserved lifetime distributions may not be smooth functions. Contributions. Using the ability of deep neural networks to model complex nonlinear relationships in the input data, our contribution is a loss function (using the p-value from a modified Kuiper nonparametric two-sample test BID28) and a backpropagation algorithm that can perform model-free (nonparametric) unsupervised clustering of subjects based on their latent lifetime distributions, even in the absence of end-of-life signals. The output of our algorithm is a trained deep neural network classifier that can (soft) assign test and training data subjects into K categories, from high-risk and to low-risk individuals. We apply our method to a large social network dataset and show that our approach is more robust than competing methods and obtains better clusters (higher C-index scores).Why deep neural networks. As with any optimization method that returns a point estimate (a set of neural network weights W in our case), our approach is subject to overfitting the training data. And because our loss function uses p-values, the optimization and overfitting have a rather negative name: p-hacking BID36. That is, the optimization is looking for a W (hypothesis) that decreases the p-value. Deep neural networks, however, are known to both overfit the training data and generalize well BID49. That is, the hypothesis (W) tends to also have small p-values in the (unseen) test data, despite overfitting in the training data (p-hacking).Outline: In section 3, we describe the traditional survival analysis concepts that assume the presence of end-of-life signals. In section 4, we define a loss function that quantifies the divergence between empirical lifetime distributions of two clusters without assuming end-of-life signals. We also provide a neural network approach to optimize said loss function. We describe the dataset used in our experiments followed by in section 5. In section 6, we describe a few methods in literature that are related to our work. Finally, we present our in section 7. In this section, we formally define the statistical framework underlying the clustering approach introduced later in this paper. We begin by defining the datasets relevant to the survival clustering task. Definition 1 (Dataset). Dataset D consists of a set of n subjects with each subject u having the following observable quantities DISPLAYFORM0, S u }, where X u are the covariates of subject DISPLAYFORM1 are the observed inter-event times (disease outbreaks, website usage), q u is the number of observed events of u, and S u is the time till censoring. Note that the two example datasets described in section 1 fit into this definition. For instance, in the social network dataset, for a particular user u, X u is a vector of her covariates (such as age, gender, etc.), Y u,i is the time between her ith and (i − 1)st activity (login, send/receive comments), and her time till censoring is given by, S u = t m − i Y u,i, where t m is the time of measurement. Next, we define the lifetime clustering problem applicable to the aforementioned datasets. Definition 2 (Clustering problem). Consider a dataset of n subjects, D, constructed as in definition 1. LetP (U k) be the latent lifetime distribution of all subjects U k = {u} that belong to cluster k ∈ {1, . . ., K}. Our goal is to find a mapping κ: X u → {1, . . ., K}, of covariates into clusters, in the set of all possible such mappings K, that maximizes the divergence d between the latent lifetime distributions of all subjects: DISPLAYFORM2 where U k (κ) is the set of users in U mapped to cluster k through κ, and d is a distribution divergence metric. κ * optimized in this fashion clusters the subjects into low-risk/high-risk groups. However, becauseP (U k) are latent distributions, we cannot directly optimize Eq.. Rather, our loss function must provide an indirect way to optimize Eq. without end-of-life signals. In what follows, we define the activity process of subjects in cluster k as a Random Marked Point Processes (RMPP).Definition 3 (Observable RMPP cluster process). Consider the k-th cluster. The RMPP is DISPLAYFORM3 is the inter-event time between the (i − 1)-st and the i-th activities, X k are the random variable representing the covariates of subjects in cluster k, S k is the time from last event until censoring at cluster k, and A k,i = 0 indicates an activity with an end-oflife signal, otherwise A k,i = 1. All these variables may be arbitrarily dependent. This definition is model-free, i.e., we will not prescribe a model for Φ k.Note that, at least theoretically, Φ k continues to evolve beyond the end-of-life signal, but this evolution will be ignored as it is irrelevant to us. The relative time of the i-th activity of a subject of cluster k, since the subject's first activity, is i ≤i Y k,i, as long as we haven't seen an end-of-life signal, i.e., i <i A k,i = 1.Definition 4 (RMPP Lifetime). The random variable that defines the lifetime of a subject of cluster k is DISPLAYFORM4 We now define censored lifetimes using Φ k.Definition 5 (RMPP Censored Lifetimes). The random variable that defines the last observed action time of a subject u of cluster k is DISPLAYFORM5 Let i (S k) be a random variable that denotes the number of events until the censoring time S k. The main challenge is not knowing when H k = T k, because we are unable to observed the end-of-life signal A k,i (S k) = 0. Clearly, this affects the decision of which subjects have been censored and which have not. Later, we introduce probability of end-of-life, p: DISPLAYFORM6, that provides a way around this challenge. In this section, we review the major concepts in survival analysis that are used in this paper. Let T u denote the lifetime of a subject u. For now, our description assumes an Oracle that provides end-of-life signals. Thus, in addition to Ψ u, we assume for each subject u and, another observable quantity, A u,i that denotes whether end-of-life has been reached at the user's ith activity. In survival applications, A u is typically used to specify if the required event did not occur until the end of study, known as right-censoring. We shall forego this assumption in subsequent sections and provide a way around the lack of these signals. Lifetime distribution & Hazard function (Oracle). Lifetime (or survival) distribution is defined as the probability that a subject u survives at least until time t, DISPLAYFORM0 where F u (t) is the cumulative distribution function of T u.In survival applications, it is typically convenient to define the hazard function, that represents the instantaneous rate of death of a subject given that she has survived until time t. The hazard function of a subject u is λ u (t) = dFu(t) Su(t), where dF u is the probability density of F u. Due to rightcensoring, we do not observe the true lifetimes of the subjects even in the presence of end-of-life signals, A u. We define the observed lifetime of subject u, H u, as the difference between the time of first event and time of last observed event, i.e., DISPLAYFORM0 provide a way to estimate the lifetime distribution for a set of subjects while incorporating the right censoring effect. The Kaplan-Meier estimates of lifetime distribution are given by, DISPLAYFORM1 where DISPLAYFORM2 ) denotes the number of subjects with end-of-life at time j, and r j = u∈D I[H u ≥ j] denotes the number of subjects at risk just prior to time j. Cox regression model BID12 ) is a widely used method in survival analysis to estimate the hazard function λ u (t) using the covariates, X u, of a subject u. The hazard function has the form, λ(t|X u) = λ 0 (t) · e {β T Xu}, where λ 0 (t) is a base hazard function common for all subjects, and β are the regression coefficients. The model assumes that the ratio of hazard functions of any two subjects is constant over time. This assumption is violated frequently in real-world datasets BID32. A near-extreme case when this assumption does not hold is shown in FIG0, where the survival curves of two groups of subjects cross each other. Survival Based Clustering Methods (Oracle). There have been relatively fewer works that perform survival based clustering. BID3 proposed a semi-supervised method for clustering survival data in which they assign Cox scores BID12 for each feature in their dataset and considered only the features with scores above a predetermined threshold. Then, an unsupervised clustering algorithm, like k-means, is used to group the individuals using only the selected features. BID17 proposed supervised sparse clustering as a modification to the sparse clustering algorithm of BID46. The sparse clustering algorithm has a modified k-means score that uses distinct weights in the feature set. Supervised sparse clustering initializes these feature weights using Cox scores BID12 and optimizes the same objective function. Both these methods assume the presence of end-of-life signals. In this paper, we consider the case when end-of-life signals are not available. We provide a loss function that quantifies the divergence between survival distributions of the clusters, and we minimize said loss function using a neural network in order to obtain the optimal clusters. Our goal is to cluster the subjects into K clusters ranging from low-risk to high-risk by keeping the empirical lifetime distributions of these K groups as different as they can be, while ensuring that the observed difference is statistically significant. In this section, we assume there are no end-of-life signals. We introduce a loss function that is based on a divergence measure between empirical lifetime distributions of two groups, and at the same time takes into account the uncertainty regarding the end-of-life of the subjects. Instead of a clear end-of-life signal, we specify a probability for each subject u that represents how likely her last observed activity coincides with her end-of-life. Definition 6 (Probability of end-of-life). Given a dataset D (Definition 1), we define a function, by an abuse of notation, p(X u, S u) → that gives a probability of end-of-life of each subject u. Divergence measures like Kullback-Leibler divergence and Earth-Mover's distance that do not incorporate the empirical nature of the given probability distributions are not appropriate for our task as they do not discourage highly imbalanced groups FIG0. This motivates the use of twosample tests that allow for the probability distributions to be empirical. Logrank test BID34 BID37 BID7 ) is commonly used to compare groups of subjects based on their hazard rates. However, the test assumes proportional hazards (section 3) and will not be able to find groups whose hazard rates are not proportional to each other FIG0. BID16 introduced Modified Kolmogorov-Smirnov (MKS) statistic that works for arbitrarily right-censored data and does not assume hazards proportionality. But MKS suffers from the same drawback as the standard Kolmogorov-Smirnov statistic, namely that it is not sensitive to the differences in the tails of the distributions. In this paper, we use p-value from the Kuiper statistic BID28 which extends the Kolmogorov-Smirnov statistic to increase the statistical power of distinguishing distribution tails BID45.Definition 7 (Optimization of Kuiper loss). Given a dataset D (Definition 1), we define a loss L(κ, p) where, by an abuse of notation, κ(X u) → is a mapping that performs soft clustering of subjects into two clusters 0 & 1 by outputting a probability of a subject belonging in cluster 0, and p(X u, S u) → is a function that gives a probability of end-of-life of a subject in D. Our goal is to obtainκ where the loss function DISPLAYFORM0 DISPLAYFORM1 returns the logarithm of a p-value from the Kuiper statistic BID38, with DISPLAYFORM2 and DISPLAYFORM3, and for k = 0, 1, DISPLAYFORM4 where DISPLAYFORM5 The following theorem states a few properties of our loss function. Theorem 1 (Kuiper loss properties). From Definition 7, consider two clusters with true lifetime distributionsP (U 1) andP (U 2). Assume an infinite number of samples/subjects. Then, the loss function defined in equation FORMULA12 has the following properties:(a) If the two clusters have distinct lifetime distributions, i.e. P (DISPLAYFORM6 (b) If the two clusters have the same stochastic process Ψ u (Definition 1), Ψ u = Ψ v, for any two subjects u and v, regardless of cluster assignments, then ∀κ, p, L(κ, p) → 0.We prove Theorem 1 in Appendix 8.1 by defining the activity process of the subjects using shifted Random Marked Point Processes. The loss defined above solves all the aforementioned issues; a) does not need clear end-of-life signals, b) use of a p-value forces sufficient number of examples in both groups, c) does not assume proportionality of hazards and works even for crossing survival curves, and d) accounts for differences at the tails. In this section, we describe the functions κ(·) and p(·) in definition 7 κ(·) gives the probability of a subject u being in cluster 0, and we define it using a neural network as follows, DISPLAYFORM0 where DISPLAYFORM1 are the weights and the biases of a neural network with L − 1 hidden layers, X u are the covariates of subject u, φ is an activation function (tanh or relU in our experiments), and σ is the softmax function. An example of a feedforward neural network that optimizes Kuiper loss is shown in FIG1.Next, we describe the form of end-of-life probability function, p(·). We make the reasonable assumption that p(·) is an increasing function of S u. For example, consider two subjects a and b, with last activities one year and one week before their respective time of censoring. Clearly, it is more likely that subject a's activity is her last one than that b's activity is her last one. In our experiments, we also assume that p(·) only depends on S u, and not on the covariates X u. Survival tasks commonly use the following naive technique to identify the end-of-life signal. They define p(·) using a step function, p(X u, S u) = 1[S u > W], where W is the size of an arbitrary window from the time of censoring (see FIG1 . However, this approach does not allow learning of the window size parameter W, and hence, the analysis can be highly coupled with the choice of W .We remedy this by choosing p(·) to be a smooth non-decreasing function of S u, parameters of which can be learnt by minimizing the loss function L(κ, p). We use the cumulative distribution function of an exponential distribution in our experiments, i.e, p(X u, S u) = 1 − e −β·Su FIG1. The rate parameter, β, is learnt using gradient descent along with the weights of the neural network. Extension to K Clusters Until now, we dealt with clustering the subjects into two groups. However, it is not hard to extend the framework for K clusters. We increase the number of units in the output layer from 2 to K. As before, a softmax function applied at the output layer gives probabilities that define a soft clustering of the subjects into K groups. These probabilities can be used to obtain the loss, L A,B, between any two groups, D A and D B.We define the total loss for K groups as the average of all the pairwise losses between individual groups and get the geometric mean of the pairwise p-values, i.e., DISPLAYFORM2 In other words, the loss L 1...K is minimized only if each of the individual p-values are low indicating that each group's lifetime distribution is different (in divergence) from every other group's lifetime distribution. Implementation We implement a feedforward neural network in Theano and use ADAM BID27 to optimize the loss L 1... K defined in equation 10. Each iteration of the optimization takes as input a batch of subjects (full batch or a minibatch), generates a value for the loss, calculates the gradients, and updates the parameters (β, DISPLAYFORM3). This is done repeatedly until there is no improvement in the validation loss. We use L2 regularization over the weights and experiment with different values for the regularization parameter. We also experiment with different neural network sizes (number of hidden layers, number of hidden units), activation functions for the hidden layers, and weight initialization techniques. We applied different deep learning techniques like batch normalization BID22 and dropout to better learn the neural network. In this paper, we analyze a large-scale social network dataset collected from Friendster. After processing 30TB of data, originally collected by the Internet Archive in June 2011, the ing network has around 15 million users with 335 million friendship links. Each user has profile information such as age, gender, marital status, occupation, and interests. Additionally, there are user comments on each other's profile pages with timestamps that indicate activity in the site. In our experiments, we only use the data up to March 2008 as Friendster's monthly active users have been significantly affected with the introduction of "new Facebook wall" BID39. From this, we only consider a subset of 1.1 million users who had participated in atleast one comment, and had specified their basic profile information like age and gender. We make our processed data available to the public at location (anonymized). We build the dataset D: DISPLAYFORM0, S u } (Definition 1) for our clustering task as follows. X u: We use each user's profile information (like age, gender, relationship status, occupation and location) as features. For the high-dimensional discrete attributes like location and occupation, we use 20 most frequently occurring values. To help improve the performance of competing methods, we also construct additional features using the user's activity over the initial 5 months (like number of comments sent and received, number of individuals interacted with, etc.). In total, we construct 60 features that are used for each of the models in our experiments. Y u,i: We define Y u,i as the time between u's ith comment (sent or received) and (i − 1)st comment (sent or received). q(u) is then defined as the total number of comments the user participated in. S u: We calculate S u as the time between u's last activity and the chosen time of measurement (March 2008). We experimented with different neural network architectures as shown in TAB1. In Table 1, we show the for a simple neural network configuration with one fully-connected hidden layer with 128 hidden units and tanh activation function. We use a batch size of 8192 and a learning rate of 10 −4. We also use batch normalization BID22 to facilitate convergence, and regularize the weights of the neural network using an L2 penalty of 0.01. Appendix 8.2 shows a more detailed evaluation of different architecture choices. We evaluate the models using 10-fold cross validation as follows. We split the dataset D randomly into 10 folds, sample 100,000 users without replacement from ith fold for testing and sample 100,000 users similarly from the remaining 9 folds for training. We use 20% of the training data as validation data for early stopping in our neural network training. We repeat this for i ranging from 1 to 10.We compare our clustering approach with the only two survival-based clustering methods in literature; a) Semi-supervised clustering BID3 ) and b) Supervised sparse clustering BID17. Since both these methods require clear end-of-life signals, we use an arbitrary window of 10 months (i.e., a pre-defined "timeout") prior to the time of measurement in order to obtain these signals FIG1. We also try window sizes of 0 months (only the users with activity at t m are censored) and 5 months, and obtain similar (not reported here). We test our approach in two cases -in the presence and lack of end-of-life signals. In the former case, we optimize the loss function in equation FORMULA2 keeping p(·) fixed to the end-of-life signals obtained from using a window of 10 months. In the latter case, our approach learns the latent end-of-life signals. We also experiment with a loss function based on the Kolmogorov-Smirnov statistic (denoted NN-KS) and report the performance for the same. We evaluate the clusters obtained from each of these methods using concordance index. Concordance Index Concordance index or C-index BID18 ) is a commonly used metric in survival applications BID1 BID33 to quantify a model's ability to discriminate between subjects with different lifetimes. It calculates the fraction of pairs of subjects for which the model predicts the correct order of survival while also incorporating censoring. We use the end-of-life signals calculated using a pre-defined "timeout" of 10 months. Rather than populating all possible pairs of users, we sample 10000 random pairs to calculate the C-index. Table 1 shows the C-index values for the baselines and the proposed method. Table 1: C-index (%) for clusters from different methods with K = 2, 3, 4 where K is the number of clusters. The proposed approach is more robust with lower values of standard deviations than the competing methods. DISPLAYFORM0 Discussion The proposed neural network approach performs better on average than the two competing methods. Even without end-of-life signals, the proposed approach achieves comparable scores for K = 3, 4 and the best C-index score for K = 2. Although NN-Kuiper is theoretically more robust than NN-KS because of its increased statistical power in distinguishing distribution tails BID45, we do not observe a performance difference in the Friendster dataset. Further, we use the endof-life signals obtained using a window of 10 months to calculate the empirical lifetime distributions of the clusters identified by the neural network (FIG2). The empirical lifetime distributions of clusters seem distinct from each other at K = 2 but not at K = 3, 4. In addition, there is not significant gain in the C-index values as we increase the number of clusters from K = 2 to K = 4. Hence, we can conclude that there are only two types of users in the Friendster dataset -long-lived and short-lived. Majority of the work in survival analysis has dealt with the task of predicting the survival outcome especially when the number of features is much higher than the number of subjects BID47 BID9 BID19 BID41. A number of approaches have also been proposed to perform feature selection in survival data BID24 BID29. In the social network scenario, BID43 tried to predict the relationship building time, that is, the time until a particular link is formed in the network. Many unsupervised approaches have been proposed to identify cancer subtypes in gene expression data without considering the survival outcome BID13 BID2 BID6. Traditional semi-supervised clustering methods BID0 BID4 BID5 BID35 do not perform well in this scenario since they do not provide a way to handle the issues with right censoring. Semi-supervised clustering BID3 and supervised sparse clustering Witten and Tibshirani (2010a) use Cox scores BID12 to identify features associated with survival. They treat these features differently in order to perform clustering based on the survival outcome. Although there are quite a few works on using neural networks to predict the hazard rates of individuals BID14, to the best of our knowledge, there hasn't been a work on using neural networks for a survival-based clustering task. Recently, Alaa and van der proposed a nonparametric Bayesian approach for survival analysis in the case of more than one competing events (multiple diseases). They not only assume the presence of end-of-life signals but also the type of event that caused the end-of-life. BID33 optimize a loss based on Cox's partial likelihood along with a penalty using a deep neural network to predict the probability of survival at a point in time. Here we considered the task of clustering subjects into low-risk/high-risk groups without observing any end-of-life signals. Extensive research has been done on what is known as frailty analysis, for predicting survival outcomes in the presence of clustered observations BID20 BID10 BID21. Although frailty models provide more flexibility in the presence of clustered observations, they do not provide a mechanism for obtaining the clusters themselves, which is our primary goal. In addition, our approach does not assume proportional hazards unlike most frailty models. In this work we introduced a Kuiper-based nonparametric loss function, and a corresponding backpropagation procedure (which backpropagates the loss over clusters rather than the loss per training example). These procedures are then used to train a feedforward neural network to inductively assign observed subject covariates into K survival-based clusters, from high-risk to low-risk subjects, without requiring an end-of-life signal. We showed that the ant neural network produces clusters with better C-index values than other competing methods. We also presented the survival distributions of the clusters obtained from our procedure and concluded that there were only two groups of users in the Friendster dataset. Both parts (a) and (b) of our proof need definition 3 that translates the observed data D u for subject u into a stochastic process. Proof of (a): If the two clusters have distinct lifetime distributions, it means that the distributions of T 0 and T 1 in eq. are different. Then, either the right-censoring δ in eq. does not allow us to see the difference between T 0 and T 1, and then there is no mappingsp andκ that can get the distribution of S 0 (t;κ,p) and S 1 (t;κ,p) to be distinct, implying an L(κ, p) → 0, as n → ∞ as the observations come from the same distribution, making the Kuiper score asymptotically equal to one; or δ does allow us to see the difference and then, clearlyp ≡ 0 with a mappingκ that assigns more than half of the subjects to their correct clusters, which would allow us to see the difference in H 0 and H 1, would give Kuiper score asymptotically equal to zero. Thus, L(κ, p) → −∞, as n → ∞.Proof of (b): Because κ only take the subject covariates as input, and there are no dependencies between the subject covariates and the subject lifetime in eq., any clustering based on the covariates will be a random assignment of users into clusters. Moreover, from eq., the censoring time of subject u, S u, has the same distribution for both clusters because the RMPPs are the same. Thus, H 0 d = H 1, i.e., H 0 and H 1 have the same distributions, and the Kuiper p-value test returns zero, L(κ, p) → 0, as n → ∞. Table 4: C-index (%) over different learning rates and batch sizes for the proposed NN approach with Kuiper loss (with learnt exponential) and K = 2.

Represent the short summary for finding the corresponding scientific article (long)

The goal of survival clustering is to map subjects into clusters. Without end-of-life signals, this is a challenging task. To address this task we propose a new loss function by modifying the Kuiper statistics.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the scientific article to find a short TLDR

Question: Bayesian optimization (BO) is a popular methodology to tune the hyperparameters of expensive black-box functions. Despite its success, standard BO focuses on a single task at a time and is not designed to leverage information from related functions, such as tuning performance metrics of the same algorithm across multiple datasets. In this work, we introduce a novel approach to achieve transfer learning across different datasets as well as different metrics. The main idea is to regress the mapping from hyperparameter to metric quantiles with a semi-parametric Gaussian Copula distribution, which provides robustness against different scales or outliers that can occur in different tasks. We introduce two methods to leverage this estimation: a Thompson sampling strategy as well as a Gaussian Copula process using such quantile estimate as a prior. We show that these strategies can combine the estimation of multiple metrics such as runtime and accuracy, steering the optimization toward cheaper hyperparameters for the same level of accuracy. Experiments on an extensive set of hyperparameter tuning tasks demonstrate significant improvements over state-of-the-art methods. Tuning complex machine learning models such as deep neural networks can be a daunting task. Object detection or language understanding models often rely on deep neural networks with many tunable hyperparameters, and automatic hyperparameter optimization (HPO) techniques such as Bayesian optimization (BO) are critical to find the good hyperparameters in short time. BO addresses the black-box optimization problem by placing a probabilistic model on the function to minimize (e.g., the mapping of neural network hyperparameters to a validation loss), and determine which hyperparameters to evaluate next by trading off exploration and exploitation through an acquisition function. While traditional BO focuses on each problem in isolation, recent years have seen a surge of interest in transfer learning for HPO. The key idea is to exploit evaluations from previous, related tasks (e.g., the same neural network tuned on multiple datasets) to further speed up the hyperparameter search. A central challenge of hyperparameter transfer learning is that different tasks typically have different scales, varying noise levels, and possibly contain outliers, making it hard to learn a joint model. In this work, we show how a semi-parametric Gaussian Copula can be leveraged to learn a joint prior across datasets in such a way that scale issues vanish. We then demonstrate how such prior estimate can be used to transfer information across tasks and objectives. We propose two HPO strategies: a Copula Thompson Sampling and a Gaussian Copula Process. We show that these approaches can jointly model several objectives with potentially different scales, such as validation error and compute time, without requiring processing. We demonstrate significant speed-ups over a number of baselines in extensive experiments. The paper is organized as follows. Section 2 reviews related work on transfer learning for HPO. Section 3 introduces Copula regression, the building block for the HPO strategies we propose in Section 4. Specifically, we show how Copula regression can be applied to design two HPO strategies, one based on Thompson sampling and an alternative GP-based approach. Experimental are given in Section 5 where we evaluate both approaches against state-of-the-art methods on three algorithms. Finally, Section 6 outlines and further developments. A variety of methods have been developed to induce transfer learning in HPO. The most common approach is to model tasks jointly or via a conditional independence structure, which has been been explored through multi-output GPs , weighted combination of GPs (; ;), and neural networks, either fully Bayesian or hybrid (; ;). A different line of research has focused on the setting where tasks come over time as a sequence and models need to be updated online as new problems accrue. A way to achieve this is to fit a sequence of surrogate models to the residuals relative to predictions of the previously fitted model . Specifically, the GP over the new task is centered on the predictive mean of the previously learned GP. Finally, rather than fitting a surrogate model to all past data, some transfer can be achieved by warm-starting BO with the solutions to the previous BO problems (; b). A key challenge for joint models is that different black-boxes can exhibit heterogeneous scale and noise levels . To address this, some methods have instead focused on search-space level, aiming to prune it to focus on regions of the hyperparameter space where good configurations are likely to lie. An example is Wistuba et al. (2015a), where related tasks are used to learn a promising search space during HPO, defining task similarity in terms of the distance of the respective data set meta-features. A more recent alternative that does not require meta-features was introduced in , where a restricted search space in the form of a low-volume hyper-rectangle or hyper-ellipsoid is learned from the optimal hyperparameters of related tasks. Rank estimation can be used to alleviate scales issues however the difficulty of feeding back rank information to GP leads to restricting assumptions, for instance does not model the rank estimation uncertainty while uses independent GPs removing the adaptivity of the GP to the current task. Gaussian Copula Process (GCP) can also be used to alleviate scale issues on a single task at the extra cost of estimating the CDF of the data. Using GCP for HPO was proposed in to handle potentially non-Gaussian data, albeit only considering non-parametric homoskedastic priors for the single-task and single objective case. For each task denote with f j: R p → R the error function one wishes to minimize, and with the evaluations available for an arbitrary task. Given the evaluations on M tasks, we are interested in speeding up the optimization of an arbitrary new task f, namely in finding arg min x∈R p f (x) in the least number of evaluations. One possible approach to speed-up the optimization of f is to build a surrogate modelf (x). While using a Gaussian process is possible, scaling such an approach to the large number of evaluations available in a transfer learning setting is challenging. Instead, we propose fitting a parametric estimate of f θ (x) distribution which can be later used in HPO strategies as a prior of a Gaussian Copula Process. A key requirement here is to learn a joint model, e.g., we would like to find θ which fits well on all observed tasks f j. We show how this can be achieved with a semi-parametric Gaussian Copula in two steps: first we map all evaluations to quantiles with the empirical CDF, and then we fit a parametric Gaussian distribution on quantiles mapped through the Gaussian inverse CDF. First, observe that as every y i comes from the same distribution for a given task, the probability integral transform in u i = F (y i), where F is the cumulative distribution function of y. We then model the CDF of (u 1, . . ., u N) with a Gaussian Copula: where Φ is the standard normal CDF and φ µ,Σ is the CDF of a normal distribution parametrized by µ and Σ. Assuming F to be invertible, we define the change of variable z = Φ −1 • F (y) = ψ(y) and g = ψ • f. 1 We regress the marginal distribution of P (z|x) with a Gaussian distribution whose mean and variance are two deterministic parametric functions given by where h w h (x) ∈ R d is the output of a multi-layer perceptron (MLP) where are projection parameters and Ψ(t) = log(1 + exp t) is an activation mapping to positive numbers. The parameters θ = {w h, w µ, b µ, w σ, b σ} together with the parameters in MLP are learned by minimizing the Gaussian negative log-likelihood on the available evaluations, e.g., by minimizing with SGD. Here, the term ψ (ψ −1 (z)) accounts for the change of variable z = ψ(y). Due to the term ψ (ψ −1 (z)), errors committed where the quantile function changes rapidly have larger gradient than when the quantile function is flat. Note that while we weight evaluations of each tasks equally, one may alternatively normalize gradient contributions per number of task evaluations. The transformation ψ requires F, which needs to be estimated. Rather than using a parametric or density estimation approach, we use the empirical CDFF (t) = 1 N N i=1 1 yi≤t. While this estimator has the advantage of being non-parametric, it leads to infinite value when evaluating ψ at the minimum of maximum of y. To avoid this issue, we use the Winsorized cut-off estimator where N is the number of observations of y and choosing δ N = 1 4N 1/4 √ π log N strikes a bias-variance trade-off . This approach is semi-parametric in that the CDF is estimated with a non-parametric estimator and the Gaussian Copula is estimated with a parametric approach. The benefit of using a non-parametric estimator for the CDF is that it allows us to map the observations of each task to comparable distributions as z j ∼ N for all tasks j. This is critical to allow the joint learning of the parametric estimates µ θ and σ θ, which share their parameter θ across all tasks. As our experiments will show, one can regress a parametric estimate that has a standard error lower than 1. This means that information can be leveraged from the evaluations obtained on related tasks, whereas a trivial predictor for z would predict 0 and yield a standard error of 1. In the next section we show how this estimator can be leveraged to design two novel HPO strategies. 4 COPULA BASED HPO 4.1 COPULA THOMPSON SAMPLING Given the predictive distribution P (z|x) ∼ N (µ θ (x), σ θ (x)), it is straightforward to derive a Thompson sampling strategy exploiting knowledge from previous tasks. Given N candidate hyperparameter configurations x 1,..., x N, we sample from each predictive distributionz i ∼ N (µ θ (x i), σ θ (x i)) and then evaluate f (x i) where i = arg min izi. Pseudo-code is given in the appendix. While this approach can re-use information from previous tasks, it does not exploit the evaluations from the current task as each draw is independent of the observed evaluations. This can become an issue if the new black-box significantly differs from previous tasks. We now show that Gaussian Copula regression can be combined with a GP to both learn from previous tasks while adapting to the current task. Instead of modeling observations with a GP, we model them as a Gaussian Copula Process (GCP) . Observations are mapped through the bijection ψ = Φ −1 • F, where we recall that Φ is the standard normal CDF and that F is the CDF of y. As ψ is monotonically increasing and mapping into the line, we can alternatively view this modeling as a warped GP with a non-parametric warping. One advantage of this transformation is that z = ψ(y) follows a normal distribution, which may not be the case for y = f (x). In the specific case of HPO, y may represent accuracy scores in of a classifier where a Gaussian cannot be used. Furthermore, we can use the information gained on other tasks with µ θ and σ θ by using them as prior mean and variance. To do so, the following residual is modeled with a GP: where g = ψ • f. We use a Matérn-5/2 covariance kernel and automatic relevance determination hyperparameters, optimized by type II maximum likelihood to determine GP hyperparameters . Fitting the GP gives the predictive distribution of the residual surrogatê Because µ θ and σ θ are deterministic functions, the predictive distribution of the surrogateĝ is then given byĝ Using this predictive distribution, we can select the hyperparameter configuration maximizing the Expected Improvement (EI) of g(x). The EI can then be defined in closed form as, with Φ and φ being the CDF and PDF of the standard normal, respectively. When no observations are available, the empirical CDFF is not defined. Therefore, we warm-start the optimization on the new task by sampling a set of N 0 = 5 hyperparameter configurations via Thompson sampling, as described above. Pseudo-code is given in Algorithm 1. Learn the parameters θ of µ θ (x) and σ θ (x) on hold-out evaluations D M by minimizing equation 1. Sample an initial set of evaluations while Has budget do Fit the GP surrogater to the observations {(x,) | (x, y) ∈ D} Sample N candidate hyperparameters x 1,..., x N from the search space Compute the hyperparameter x i where i = arg max i EI(In addition to optimizing the accuracy of a black-box function, it is often desirable to optimize its runtime or memory consumption. For instance, given two hyperparameters with the same expected error, the one requiring fewer resources is preferable. For tasks where runtime is available, we use both runtime and error objectives by averaging in the transformed space, e.g., we set, where z error (x) = ψ(f error (x)) and z time (x) = ψ(f time (x)) denote the transformed error and time observations, respectively. This allows us to seamlessly optimize for time and error when running HPO, so that the cheaper hyperparameter is favored when two hyperparameters lead to a similar expected error. Notice many existing multi-objective methods can potentially be combined with our Copula transformation as an extension, which we believe is an interesting venue for future work. We considered the problem of tuning three algorithms on multiple datasets: XGBoost , a 2-layer feed-forward neural network (FCNET) , and the RNN-based time series prediction model proposed in (DeepAR). We tuned XGBoost on 9 libsvm datasets to minimize 1−AUC, and FCNET on 4 datasets from to minimize the test mean squared error. As for DeepAR, the evaluations were collected on the data provided by GluonTS , consisting of 6 datasets from the M4-competition and 5 datasets used in , and the goal is to minimize the quantile loss. Additionally, for DeepAR and FCNET the runtime to evaluate each hyperparameter configuration was available, and we ran additional experiments exploiting this objective. More details on the HPO setup are in Table 1, and the search spaces of the three problems is in Table 4 of the appendix. Lookup tables are used as advocated in , more details and statistics can be found in the appendix. We compare against a number of baselines. We consider random search and GP-based BO as two of the most popular HPO methods. As a transfer learning baseline, we consider warm-start GP , using the best-performing evaluations from all the tasks to warm start the GP on the target task (WS GP best). As an extension of WS GP best, we apply standardization on the objectives of the evaluations for every task and then use all of them to warm start the GP on the target task (WS GP all). We also compare against two recently published transfer learning methods for HPO: ABLR and a search space-based transfer learning method . ABLR is a transfer learning approach consisting of a shared neural network across tasks on top of which lies a Bayesian linear regression layer per task. transfers information by fitting a bounding box to contain the best hyperparameters from each previous task, and applies random search (Box RS) or GP-based BO (Box GP) in the learned search space. We assess the transfer learning capabilities of these methods in a leave-one-task-out setting: we sequentially leave out one dataset and then aggregate the for each algorithm. The performance of each method is first averaged over 30 replicates for one dataset in a task, and the relative improvements over random search are computed on every iteration for that dataset. The relative improvement for an optimizer (opt) is defined by (y random − y opt)/y random, which is upper bounded by 100%. Notice that all the objectives y are in R +. By computing the relative improvements, we can aggregate across all datasets for each algorithm. Finally, for all copula-based methods, we learn the mapping to copulas via a 3-layer MLP with 50 units per layer, optimized by ADAM with early-stopping. To give more insight into the components of our method, we perform a detailed ablation study to investigate the choice of the MLP and compare the copula estimation to simple standardization. Choice of copula estimators For copula-based methods, we use an MLP to estimate the output. We first compare to other possible options, including a linear model and a k-nearest neighbor estimator in a leave-one-out setting: we sequentially take the hyperparameter evaluations of one dataset as test set and use all evaluations from the other datasets as a training set. We report the RMSE in Table 5 of the appendix when predicting the error of the blackbox. From this table, it is clear that MLP tends to be the best performing estimator among the three. In addition, a low RMSE indicates that the task is close to the prior that we learned on all the other tasks, and we should thus expect transfer learning methods to perform well. As shown later by the BO experiments, FCNET has the lowest RMSE among the three algorithms, and all transfer learning methods indeed perform much better than single-task approaches. Homoskedastic and Heteroskedastic noise The proposed Copula estimator (MLP) uses heteroskedastic noise for the prior. We now compare it to a homoskedastic version where we only estimate the mean. The are summarized in Table 2 where average relative improvements over random search across all the iterations and replicates are shown. It is clear that heteroskedasticity tends to help on most datasets. Copula transformation and standardization In our method, we map objectives to be normally distributed in two steps: first we apply the probability integral transform, followed by a Copula transform using the inverse CDF of a Gaussian. To demonstrate the usefulness of such transformation, we compare it to a simple standardization of the objectives where mean and std are computed on each datasets separately. Results are reported in Table 2. It is clear that standardization performs significantly worse than the Copula transformation, indicating that it is not able to address the problem of varying scale and noise levels across tasks. Note that the relative improvement objective is not lower bounded, so that when random search finds very small values the scale of relative improvement can be arbitrary large (such as for the Protein dataset in FCNET). Table 2: Relative improvements over random search. TS std and GP std respectively using a simple standardization instead of the copula transformation. Ho and He stand for Homoskedastic and Heteroskedastic noise. We now compare the proposed methods to other HPO baselines. The on using only the error information are shown first followed by the using both time and error information. Results using only error information We start by studying the setting where only error objectives are used to learn the copula transformation. Within each task, we first aggregate 30 replicates for each method to compute the relative improvement over random search at every iteration, and then average the across all iterations. The are reported in Table 3, showing that CGP is the best method for almost every task except XGBoost. In XGBoost, there are several tasks on which methods without transfer learning perform quite well. This is not surprising as we observe in an ablation study on copula estimators (see Table 5 in the appendix) that some tasks in XGBoost have relatively high test errors, implying that the transferred prior will not help. In those tasks, CGP is usually the runner-up method after standard GP. We also report the at iteration 10, 50 and 100 in the Tables 7, 8 and 9 in the appendix where we observe CGP and Box RS are the most competitive methods at 10th iteration but at 100 iteration, CGP is clearly the best transfer learning method. This highlights the advantage of being adaptive to the target task of our method while making effective transfer in the beginning. We also show on two example datasets from each algorithm in Figure 1, reporting confidence intervals obtained via bootstrap. Note that the performance of the methods in the examples for DeepAR and XGBoost exhibit quite high variations, especially in the beginning of the BO. We conjecture this is due to an insufficient number of evaluations in the lookup tables. Nevertheless, the general trend is that CTS and CGP outperform all baselines, especially in the beginning of the BO. It can also be observed that CGP performs at least on par with the best method at the end of the BO. Box RS is also competitive at the beginning, but as expected fails to keep its advantage toward the end of the BO. Results using both error and time information We then studied the ability of the copula-based approaches to transfer information from multiple objectives. Notice it is possible to combine Copula transformation with other multi-objective BO methods and we will leave this direction as future work. We show two example tasks in DeepAR and FCNET in Figure 2, where we fix the total number of iterations and plot performance against time with 2 standard error. To obtain distributions over seeds for one method, we only consider the time range where 20 seeds are available,which explains why methods can start and end at different times. With the ability to leverage training time information, the copula-based approaches have a clear advantage over all baselines, especially at the beginning of the optimization. We also report aggregate performance over all the tasks in Table 6 in the appendix. Due to the different methods finishing the optimization at different times, we only compare them up to the time taken by the fastest method. For each method we first compute an average over 30 replicates, then compute the relative improvement over random search, and finally average the across all time points. The copula based methods converge to a good hyperparameter configuration significantly faster than all the considered baselines. Note that we obtain similar as for Hyperband-style methods , where the optimization can start much earlier than standard HPO, with the key difference that we only require a single machine. We introduced a new class of methods to accelerate hyperparameter optimization by exploiting evaluations from previous tasks. The key idea was to leverage a semi-parametric Gaussian Copula prior, using it to account for the different scale and noise levels across tasks. Experiments showed that we considerably outperform standard approaches to BO, and deal with heterogeneous tasks more robustly compared to a number of transfer learning approaches recently proposed in the literature. Finally, we showed that our approach can seamlessly combine multiple objectives, such as accuracy and runtime, further speeding up the search of good hyperparameter configurations. A number of directions for future work are open. First, we could combine our Copula-based HPO strategies with Hyperband-style optimizers . In addition, we could generalize our approach to deal with settings in which related problems are not limited to the same algorithm run over different datasets. This would allow for different hyperparameter dimensions across tasks, or perform transfer learning across different black-boxes. Learn the parameters θ of µ θ (x) and σ θ (x) on hold-out evaluations D M by minimizing equation 1. while Has budget do Sample N candidate hyperparameters x 1,..., x N from the search space where i = arg min izi end while To speed up experiments we used a lookup table approach advocated in which proposed to use an extrapolation model built on pre-generated evaluations to limit the number of blackbox evaluations, thus saving a significant amount of computational time. However, the extrapolation model can introduce noise and lead to inconsistencies compared to using real blackbox evaluations. As a , in this work we reduced BO to the problem of selecting the next hyperparameter configurations from a fixed set that has been evaluated in advance, so that no extrapolation error is introduced. All evaluations were obtained by querying each algorithm at hyperparameters sampled (log) uniformly at random from their search space as described in Table 4. The CDF on the error objectives is given in Figure 3. Results on different iterations. We plot the improvement over random research for all the methods at iteration 10, 50 and 100 at Table 7, 8 and 9, respectively. In short, at 10th iteration, transfer learning methods, especially our CGP and Box RS, performed much better than GP. But, when looking at at 50 and 100 iterations, CGP outperforms clearly all other transfer methods because of its improved adaptivity. More details on prior MLP architecture. The MLP used to regress µ θ and σ θ consists of 3 layers with 50 nodes, each with a dropout layer set to 0.5. The learning rate is set to 0.01, batch size to 64 and we optimize over 100 gradient updates 3 times, lowering the learning rate by 10 each time. Table 9: Relative improvements over random search at iteration 100.

represent this text to find a much much longer article about it with say hundreds-thousands of words

We show how using semi-parametric prior estimations can speed up HPO significantly across datasets and metrics.

represent the short summary for finding the corresponding scientific article (long)

We propose a new class of inference models that iteratively encode gradients to estimate approximate posterior distributions.