Question: Represent the article for finding a single-sentence TLDR

Question: Massively multi-label prediction/classification problems arise in environments like health-care or biology where it is useful to make very precise predictions. One challenge with massively multi-label problems is that there is often a long-tailed frequency distribution for the labels, ing in few positive examples for the rare labels. We propose a solution to this problem by modifying the output layer of a neural network to create a Bayesian network of sigmoids which takes advantage of ontology relationships between the labels to help share information between the rare and the more common labels. We apply this method to the two massively multi-label tasks of disease prediction (ICD-9 codes) and protein function prediction (Gene Ontology terms) and obtain significant improvements in per-label AUROC and average precision. In this paper, we study general techniques for improving predictive performance in massively multilabel classification/prediction problems in which there is an ontology providing relationships between the labels. Such problems have practical applications in biology, precision health, and computer vision where there is a need for very precise categorization. For example, in health care we have an increasing number of treatments that are only useful for small subsets of the patient population. This forces us to create large and precise labeling schemes when we want to find patients for these personalized treatments. One large issue with massively multi-label prediction is that there is often a long-tailed frequency distribution for the labels with a large fraction of the labels having very few positive examples in the training data. The corresponding low amount of training data for rare labels makes it difficult to train individual classifiers. Current multi-task learning approaches enable us to somewhat circumvent this bottleneck through sharing information between the rare and cofmmon labels in a manner that enables us to train classifiers even for the data poor rare labels BID6.In this paper, we introduce a new method for massively multi-label prediction, a Bayesian network of sigmoids, that helps achieve better performance on rare classes by using ontological information to better share information between the rare and common labels. This method is based on similar ideas found in Bayesian networks and hierarchical softmax BID18. The main distinction between this paper and prior work is that we focus on improving multi-label prediction performance with more complicated directed acyclic graph (DAG) structures between the labels while previous hierarchical softmax work focuses on improving runtime performance on multi-class problems (where labels are mutually exclusive) with simpler tree structures between the labels. In order to demonstrate the empirical predictive performance of our method, we test it on two very different massively multi-label tasks. The first is a disease prediction task where we predict ICD-9 (diagnoses) codes from medical record data using the ICD-9 hierarchy to tie the labels together. The second task is a protein function prediction task where we predict Gene Ontology terms BID0 BID5 from sequence information using the Gene Ontology DAG to combine the labels. Our experiments indicate that our new method obtains better average predictive performance on rare labels while maintaining similar performance on common labels. The goal of multi-label prediction is to learn the distribution P (L|X) which gives the probability of an instance X having a label L from a dictionary of N labels. We are particularly interested in the case where there is an ontology providing superclass relationships between the labels. This ontology consists of a DAG where every label L is a node and every directed edge from L i to L j indicates that the label L i is a superclass of the label L j. FIG0 gives corresponding example simplified subgraphs from both the ICD-9 hierarchy and the Gene Ontology DAG. We define parents(L) to be the direct parents of L. We define ancestors(L) to be all of the nodes that have a directed path to L. Gene Ontology The classical approach for solving this problem is to learn separate functions for each label. This transforms the problem into N binary prediction problems which can each be solved with standard techniques. The main issue with this approach is that it is less sample efficient in that it does not share information between the labels. A more sophisticated approach is to use multi-task learning techniques to share information between the individual label-specific binary classifiers. One approach for doing this with neural networks is to introduce shared layers between the different binary classifiers. The ing output layer is a flat structure of sigmoid outputs, with each sigmoid output representing one P (L|X). This reduces the number of parameters needed for every label and allows information to be shared among the labels BID6. However, even with this weight sharing, the final output layer still needs to be learned independently for each label. We propose a modification of the output layer by constructing a Bayesian network of sigmoids in order to use the ontology to share additional information between labels in a more guided way. The general idea is that we assume that the probability of our labels follows a Bayesian network BID19 with each edge in the ontology representing an edge within the Bayesian network. This, along with the fact that the edges denote superclasses, enables us to factor the probability of a label into several conditional probabilities. DISPLAYFORM0 As the edges denote superclasses, having a child label implies having every ancestor DISPLAYFORM1 From Baysian network assumption on the subgraph consisting of L and ancestors(L) BID19 We are now able to learn the conditional probability distributions P (L|X, parents(L)) for every label in the ontology and use the above formula to reconstruct the final target probabilities P (L|X). Consider the example simplified ICD-9 graph in FIG0. For this graph, we would learn P (Cancer|X), P (LungCancer|Cancer, X), and P (SkinCancer|Cancer, X). We would then be able to compute P (LungCancer|X) = P (Cancer|X) × P (LungCancer|Cancer, X).The intuition of why this factoring might be useful is that it enables the transferring of knowledge from more common higher-level labels to more rare lower-level labels. Consider the case where L is very rare. In that case it is difficult to learn P (L|X) directly due to the small amount of training data. However, the decomposed version ∈{L}∪ancestors(L) P (|X, parents) includes classifiers from the ancestors of L that have more training data and might be easier to learn. This factoring allows additional signal from the better trained higher-level labels to feed directly into the probability computation for the rare leaf L. If we can rule out one of the higher-level labels, we can also rule out a lower-level label. For example, consider the ICD-9 graph illustrated in FIG0. We might not have enough patients with lung cancer to directly learn an optimal P (LungCancer|X). However, we can pool all of our cancer patients to learn a hopefully more optimal P (Cancer|X). We can then use our Bayesian network factoring to incorporate the better trained P (Cancer|X) classifier in our calculation for P (LungCancer|X). In our experiments we show that this intuition plays out in practice through improved performance on rare labels. The Bayesian network assumption plays an important role in allowing us to factor the probabilities in this manner. In order to perform our factoring, we must assume that every subgraph of the ontology consisting of the nodes {L} ∪ ancestors(L) correctly represents a Bayesian network for the label probability distribution. These subgraphs are only correct Bayesian networks if the probability of every label L is conditionally independent of the probabilities of non-descendent labels given the parent labels and X BID20 ). This might seem somewhat limiting, but there are two reasons why this assumption is weaker than it might appear. First, we only require a Bayesian network to be correct for the subgraphs of the form {L} ∪ ancestors(L). This is true because we only consider the nodes {L} ∪ ancestors(L) when we do our factoring. This is a significantly weaker assumption than requiring the entire graph to follow a Bayesian network. One direct application of this is that every tree ontology can meet this assumption. The proof for this is that every {L} ∪ ancestors(L) subgraph of a tree is a simple chain. A simple chain is not able to violate the conditional independence assumption behind Bayesian networks because it has no non-descendent nodes that are not already ancestors. Ancestor nodes are always conditionally independent with the label given the parents because the edges represent superclasses and thus either the ancestors are always present if the parent i present or the label is always not present if the parent is not present. The second reason why this assumption is weaker than it might appear is that we only require conditional independence given a particular instance X. As an illustrative example, consider the two ICD-9 labels of male breast cancer (ICD-9 175) and female breast cancer (ICD-9 174). Male breast cancer and female breast cancer are trivially not conditionally independent due to the gender qualifier making them mutually exclusive. However, male breast cancer and female breast cancer become conditionally independent once you condition on the gender of the patient. Thus conditioning on the exact instance X enables more conditional independence than would otherwise be available. Nevertheless, even with these caveats, there will be some circumstances in which this conditional independence assumption is violated. In these situations, our factoring is not valid and our computed product ∈{L}∪ancestors(L) P (|X, parents) might diverge from the actual P (L|X). Yet, even in these situations, the ing scores can still be empirically useful. We demonstrate that this is the case in our experiments by showing performance improvements in a protein function prediction task that almost assuredly violates this conditional independence assumption. There are many potential ways in which the conditional probabilities P (L|X, parents(L)) could be modeled. We exclusively focus on modeling these probabilities using a sigmoid function computed on logits from neural networks. We define an encoder neural network for every task that takes in the input X and returns a fixed-length representation of the input. We also define a fixed-length embedding for every label L by constructing an output embedding matrix such that e L is the embedding for L. This encoder and label embedding then allow us to model P (L|X, parents(L)) as σ(encoder(X) · e L ), where σ indicates the sigmoid function and · indicates a dot product. Note that parents(L) is not used in this formula. This is because there is a unique set of parents for every label L, so there is no need to have distinct e L vectors for different sets of parents. We can then train P (L|X, parents(L)) by using cross entropy loss on patients who have all the labels in parents(L). Note that we explicitly do not train each of the conditional probabilities on every patient. We can only train the conditional probabilities on patients who satisfy the conditional requirement of having the parent labels. This does not change the number of positive examples for each classifier, but it does significantly reduce the number of negative examples for the lower level classifiers. For example, consider the ICD-9 subgraph shown in FIG0. In this situation, we have three labels and thus need to learn three conditional probabilities: P (Cancer|X), P (LungCancer|Cancer, X) and P (BreastCancer|Cancer, X). We have three labels, so our label embedding matrix consists of e Cancer, e LungCancer and e BreastCancer. We can now compute P (LungCancer|X) and P (BreastCancer|X) as follows: DISPLAYFORM0 As a baseline, we also train models with a normal flat sigmoid output layer. In these models we directly learn P (L|X) for each label. Similar to the conditional probabilities, we can define these probabilities as a sigmoid of the output from a neural network. We define P (L|X) to be σ(encoder(X) · e L ). We can then train P (L|X) using cross entropy loss on all patients. We evaluated the predictive performance of our method on two very different massively multi-label problems. We consider the task of predicting future diseases for patients given medical history in the form of ICD-9 codes and the task of predicting protein function from sequence data in the form of Gene Ontology terms. In this section, we introduce the datasets, encoders and baselines used for each problem.3.1 DISEASE PREDICTION 3.1.1 PROBLEM One of our experiments consists of predicting diseases in the form of ICD-9 codes from electronic medical record (EMR) data. We have two years and nine months of data covering 2013, 2014, and the first nine months of 2015. We use two years of history to predict which ICD-9 codes will appear in the following nine months. The problem setup for this experiment closely matches the setup in BID17. We use a large insurance claims dataset from [redacted to preserve anonymity] for modeling. Our claims data consists of diagnoses (ICD-9), medications (NDC), procedures (CPT), and some metadata such as age, gender, location, general occupation, and employment status. We restrict our analysis to patients who were enrolled during 2013, 2014 and January 2015.We have 15.7 million patients, of which a random 5% are used for validation and 5% are used for testing. This dataset is quite large, much larger than what is usually available in a hospital. Thus we consider two cases of this problem. The "high data case" is where we use all remaining 14.1 million patients for training. The " low data case" consists of training with a 2% random sample of 281,874 patients and is much closer in size to normal hospital datasets BID8 BID1.Our target label dictionary for this task consists of all leaf ICD-9 billing codes that appear at least 5 times in the training data. We only predict leaf codes as those are the only codes allowed for billing and thus the only ICD-9 codes that records are annotated with. This in a dictionary of 6,902 codes for the small disease prediction task and 12,533 codes for the large disease prediction task. We use the ICD-9 hierarchy included in the 2018AA UMLS release BID2 in order to construct relationships between the labels for our method. We additionally use the CPT and ATC ontologies included in the 2018AA for our encoder. The partitioning of the patient timelines into input history and output prediction labels as well as the subpartioning of the input history into time-bins. Each tick on the x-axis represents one month. The first two years of information is used as input and the final nine months is used to generate output prediction labels. These first two years are subdivided into six bins of the following lengths for featurization: one year, six months, three months, one month, one month, and one month. For our encoder, we use a feed-forward architecture inspired by BID1. As in their model, we split our two years of data into time-sliced bins. For each time slice, we find all the ICD-9, NDC and CPT codes that the patient experienced during the time slice. FIG1 details the exact layout of each time bin. We also add a feature for every higher-level code in the ICD-9, ATC and CPT ontologies that indicates whether the patient had any of the descendants of that particular code within the time slice. This expanded rollup scheme is structurally very similar to the subword method introduced in BID3. The weights for these input embeddings are tied to the output embedding matrix used in our output layers. We summarize the set of embeddings for each time bin using mean pooling. We also construct mean embedding for the metadata by feeding the metadata entries through an embedding matrix followed by mean pooling. Finally, we concatenate the means from each timeslice with the mean embeddings from the metadata and feed the ing vector into a feedforward neural network to compute a final patient embedding. These neural network models are trained with the Adam optimizer. The hyperparameters such as the learning rate, layer size, non-linearity, and number of layers are optimized using a grid search on the validation set. Appendix A.1 has details on the space searched as well as the best hyper-parameters for both the normal sigmoid and Bayesian network sigmoid models. Finally, as a further baseline, we also train logistic regression models individually for several rare ICD-9 codes. These models are trained on a binary matrix where each row represents a patient and each column represents an ICD-9 code, NDC code, CPT code, or metadata element. A particular row and column element is set to 1 whenever a patient has that particular item in the metadata or during the two years of provided medical history. These logistic regression models are regularized with L2 with a lambda optimized using cross-validation. One particular issue with training individual models on rare codes is that the dataset is distinctly unbalanced with vastly more negative examples than positive examples. We deal with this issue by subsampling negative examples so that the ratio of positive and negative samples is 1:10. For our other experiment, we predict protein functions in the form of Gene Ontology (GO) terms from sequence data. We focus only on human proteins that have at least one Gene Ontology annotation. Our features consist of amino acid sequences downloaded from Uniprot on July 27, 2018 . For our labels, we use the human GO labels which were generated on June 18, 2018. After joining the labels with the sequence data, we have a total of 15,497 annotated human protein sequences. A random 80% of the sequences are used for training, 10% are using for validation, and a final 10% are used for final testing. In this task we predict all leaf and higher level GO terms that appear at least 5 times in the training data. This in a target dictionary of 7,751 terms. We construct relationships between these labels using the July 24, 2018 release of the GO basic ontology. We use's 1-D CNN based encoder to encode our protein sequence information. We treat every letter in the alphabet as a word and encode each of those letters with an embedding size of size 26. We then apply a 1-D convolution with a window size of 8 over the embedded sequence. A fixed-length representation of the protein is then obtained by doing max-over-time pooling. This representation is finally fed through a ReLU and one fully connected layer. The ing fixed dimension vector is the encoded protein. For regularization, we add dropout before the convolution and fully connected layer. Following previous work, we also consider generating features using sequence alignment BID14. We use version 2.7.1 of the BLAST tool to find the most similar training set protein for every protein in our dataset . We then use this most similar protein to augment our protein encoder by adding a binary feature which signifies if the most similar protein has the particular term we are predicting. These CNN models are trained with Adam. Hyperparameters such as learning rate, number of filters, dropout, and the size of the final layer are optimized using a grid search on the validation set. See Appendix A.1 for a full listing of the space searched as well as the best hyperparameters for both the flat sigmoid and Bayesian network of sigmoids models. As a further baseline, we also consider using the BLAST features alone for predicting protein function. This model simply consists of a 1 if the most similar protein has the target term or a 0 otherwise. For these protein models, we also consider one final baseline where we take our flat sigmoid model and weight labels according to the inverse square root of their frequency. This weighting scheme is based off the subsampling scheme from BID16. Unfortunately, this baseline did not seem to perform well on rare words so we did not consider it for the disease case and our more general analysis. The for this baseline can be found in Appendix A.3. Figure 3 shows frequency binned per-label area under the receiver operating characteristic (AUROC) and average precision (AP) for less frequent labels that have at most 1,000 positive examples. See Appendix A.2 for the exact numerical which include 95 % bootstrap confidence intervals generated through 500 bootstrap samples of the test set. As shown in Figure 4, these less frequent labels cover a majority of the labels within each dataset. Our indicate that the Bayesian network of sigmoid output layer has better AUROC and average precision for rare labels in all three tasks, with the effect diminishing with increasing numbers of positive labels. This effect is especially strong in the average precision space. For example, the Bayesian network of sigmoid models obtain 187%, 28.5% and 17.9% improvements in average precision for the rarest code bin (5-10 positive examples) over the baseline models for the small disease, large disease and protein function tasks, respectively. This improvement persists for the next rarest bin (11-25 positive examples), but decreases to 89.2%, 10.7% and 11.1%. This matches our previous intuition as there is no need to transfer information from more general labels if there is enough data to model P (L|X) directly. TAB0 compares micro-AUROC and micro-AP on all labels for all three tasks. The benefits of the Bayesian sigmoid output layer seem much more limited and task specific in this setting. We do not expect significantly better in the micro averaged performance case because the micro are more dominated by more frequent codes and the Bayesian network of sigmoids is only expected to help when P (L|X) does not have enough data to be modeled directly. The Bayesian network of sigmoids output layer provides better AUROC and AP for the disease prediction task, but suffers from worse performance in the protein function task. One possible explanation for this discrepancy is that our Bayesian network assumption is guaranteed to be correct in the disease prediction task due to the tree structure of the ontology, but might not be correct in the protein function task with its more complicated DAG ontological structue. It is possible that minor violations of the Bayesian network assumption in the protein function prediction task cause the overall performance to be worse on the more common code compared to the flat sigmoid decoder. There is related work on improved softmax variants, predicting ICD-9 codes, predicting Gene Ontology terms and combining ontologies with Bayesian networks. Improved softmax variants. There has been a wide variety of work focusing on trying to come up with improved softmax variants for use in massively multi-class problems such as language modeling. This prior work primarily differs from this work in that it focuses exclusively on the multi-class case with a tree structure connecting the labels. Multi-class is distinct from multi-label in that multi-class requires each item to only have one label while multi-label allows multiple labels per item. Most of this work focuses around trying to improve the training time for the expensive softmax operation found in multi-class problems such as large-vocabulary language modeling. The most related of these variants fall under the hierarchical softmax family. Hierarchical softmax from BID18 (and related versions such as class based softmax from BID10 and adaptive softmax from BID11) focuses on speeding up softmax by using a tree structure to decompose the probability distribution. Disease prediction. Previous work has also explored the task of disease prediction through predicting ICD-9 codes from medical record data BID17 BID8 BID7. GRAM from is a particularly relevant instance which uses the CCS hierarchy to improve the encoder, ing in better predictions for rare codes. Our work differs from GRAM in that we improve the output layer while GRAM improves the encoder. Protein function prediction. Protein function prediction in the form of Gene Ontology term prediction has been considered by previous work BID14 BID15 BID4. DeepGO from BID14 is the most similar to the approach taken by this paper in that it uses a CNN on the sequence data to predict Gene Ontology terms. It also uses the ontology in that it creates a multi-task neural network in the shape of the ontology. Our work differs from DeepGO in that we focus on the rarer terms and we only modify the output layer. Combining ontologies with Bayesian networks. Phrank from is an algorithm for computing similarity scores between sets of phenotypes for use in diagnosing genetic disorders. Like this paper, Phrank constructs a Bayesian network based on an ontology. This work differs from Phrank in that we focus on the supervised prediction task of modeling the probability of a label given an instance while Phrank focuses on the simpler task of modeling the unconditional probability of a label (or set of labels). This paper introduces a new method for improving the performance of rare labels in massively multi-label problems with ontologically structured labels. Our new method uses the ontological relationships to construct a Bayesian network of sigmoid outputs which enables us to express the probability of rare labels as a product of conditional probabilities of more common higher-level labels. This enables us to share information between the labels and achieve empirically better performance in both AUROC and average precision for rare labels than flat sigmoid baselines in three separate experiments covering the two very different domains of protein function prediction and disease prediction. This improvement in performance for rare labels enables us to make more precise predictions for smaller label categories and should be applicable to a variety of tasks that contain an ontology that defines relationships between labels. This section has been redacted to preserve anonymity. A.1 HYPERPARAMETER GRID AND BEST HYPERPARAMETERS

represent the tldr for finding the corresponding scientific article

To give you a sense - "A new neural architecture where top dense layers of standard convolutional architectures are replaced with an approximation of a kernel function by relying on the Nyström approximation." should be close to "Recent work has focused on combining kernel methods and deep learning. With this in mind, we introduce Deepström networks -- a new architecture of neural networks which we use to replace top dense layers of standard convolutional architectures with an approximation of a kernel function by relying on the Nyström approximation. Our approach is easy highly flexible. It is compatible with any kernel function and it allows exploiting multiple kernels. We show that Deepström networks reach state-of-the-art performance on standard datasets like SVHN and CIFAR100. One benefit of the method lies in its limited number of learnable parameters which make it particularly suited for small training set sizes, e.g. from 5 to 20 samples per class. Finally we illustrate two ways of using multiple kernels, including a multiple Deepström setting, that exploits a kernel on each feature map output by the convolutional part of the model. Kernel machines and deep learning have mostly been investigated separately. Both have strengths and weaknesses and appear as complementary family of methods with respect to the settings where they are most relevant. Deep learning methods may learn from scratch relevant features from data and may work with huge quantities of data. Yet they actually require large amount of data to fully exploit their potential and may not perform well with limited training datasets. Moreover deep networks are complex and difficult to design and require lots of computing and memory resources both for training and for inference. Kernel machines are powerful tools for learning nonlinear relations in data and are well suited for problems with limited training sets. Their power comes from their ability to extend linear methods to nonlinear ones with theoretical guarantees. However, they do not scale well to the size of the training datasets and do not learn features from the data. They usually require a prior choice of a relevant kernel amongst the well known ones, or even require defining an appropriate kernel for the data at hand. Although most research in the field of deep learning seems to have evolved as a "parallel learning strategy" to the field of kernel methods, there are a number of studies at the interface of the two domains which investigated how some concepts can be transferred from one field to another. Mainly, there are two types of approaches that have been investigated to mix deep learning and kernels. Few works explored the design of deep kernels that would allow working with a hierarchy of representations as the one that has been popularized with deep learning (2; 14; 7; 6; 20; 23). Other studies focused on various ways to plug kernels into deep networks (13; 24; 5; 12; 25). This paper follows this latter line of research, it focuses on convolutional networks. Specifically, we propose Deepström networks which are built by replacing dense layers of a convolutional neural network by an adaptive approximation of a kernel function. Our work is inspired from Deep Fried Convnets which brings together convolutional neural networks and kernels via Fastfood, a kernel approximation technique based on random feature maps. We revisit this concept in the context of Nyström kernel approximation BID21. One key advantage of our method is its flexibility that enables the use of any kernel function. Indeed, since the Nyström approximation uses an explicit feature map from the data kernel matrix, it is not restricted to a specific kernel function and not limited only to RBF kernels, as in Fastfood approximation. This is particularly useful when one wants to use or learn multiple different kernels instead of a single kernel function, as we demonstrate here. In particular we investigate two different ways of using multiple kernels, one is a straightforward extension to using multiple kernels while the second is a multiple Deepström variant that exploits a Nyström kernel approximation for each of the feature map output by the convolutional part of the neural network. Furthermore the specific nature of our architecture makes it use only a limited number of parameters, which favours learning with small training sets as we demonstrate on targeted experiments. Our experiments on four datasets (MNIST, SVHN, CIFAR10 and CIFAR100) highlight three important features of our method. First our approach compares well to standard approaches in standard settings (using ful training sets) while requiring a reduced number of parameters compared to full deep networks and of the same order of magnitude as Deep Fried Convnets. This specific feature of our proposal makes it suitable for dealing with limited training set sizes as we show by considering experiments with tens or even fewer training samples per class. Finally the method may exploit multiple kernels, providing a new tool with which to approach the problem of multiple kernel learning (MKL), and enabling taking into account the rich information in multiple feature maps of convolution networks through multiple Deepström layers. The rest of the paper is organized as follows. We provide on kernel approximation via the Nyström and the random Fourier features methods and describe the Deep Fried Convnet architecture in Section 2. The detailed configuration of the proposed Deepström network is described in Section 3. We also show in Section 3 how Deepström networks can be used with multiple kernels. Section 4 reports experimental on MNIST, SVHN, CIFAR10 and CIFAR100 datasets to first provide a deeper understanding of the behaviour of our method with respect to the choice of the kernels and the combination of these, and second to compare it to state of the art baselines on classification tasks with respect to accuracy and to complexity issues, in particular in the small training set size setting. Kernel approximation methods have been proposed to make kernel methods scalable. Two popular methods are Nyström approximation BID21 and random features approximation BID15. The former approximates the kernel matrix by an efficient low-rank decomposition, while the latter is based on mapping input features into a low-dimensional feature space where dot products between features approximate well the kernel function. Nyström approximation BID21 It computes a low-rank approximation of the kernel matrix by randomly subsampling a subset of instances. Let consider a training set of n training samples, DISPLAYFORM0 which is selected from the training set. Assuming the subset includes the first samples, or rearranging the training samples this way, K may be rewritten as: DISPLAYFORM1 where K 11 is the Gram matrix on subset L. Nyström approximation is obtained as follows DISPLAYFORM2 From this approximation the Nyström nonlinear representation of a single example x is given by DISPLAYFORM3 where DISPLAYFORM4 Random features approximation It computes a low-dimensional feature mapφ of dimension q such that φ (·),φ(·) =k(·, ·) k(·, ·). Two instances of this method are Random Kitchen Sinks (RKS) and Fastfood (17; 9). RKS approximates a Radial Basis Function (RBF) kernel using a random feature map defined as DISPLAYFORM5 where z ∈ R p, Q ∈ R q×p and Q i,j are drawn randomly. If Q i,j are drawn according to a Gaussian distribution then the method is shown to approximate the Gaussian kernel, i.e. φ rks (DISPLAYFORM6 where σ is the hyper-parameter of the kernel. Note that σ is related to the parameters of the Gaussian distribution that generate the random features. The Fastfood method is a variant of RKS with reduced computational cost for the Gaussian kernel. It is based on approximating the matrix Q in Eq. 2, when q = p, by a product of diagonal and hadamard matrices according to DISPLAYFORM7 where S,G and B are diagonal matrices of size p × p, Π ∈ {0, 1} p×p is a random permutation matrix, H is a Hadamard matrix which does not requite to be stored, and σ is an hyperparameter. Matrix V may be used in place of Q in Eq. 2 to define the Fastfood nonlinear representation map DISPLAYFORM8 Note that this definition requires p to be a power of 2 to take advantage of the recursive structure of the Hadamard matrix. Note also that to reach a representation dimension q > p one may compute multiple V and concatenate the corresponding φ f f.Deep Fried Convnets Our attention in this work is especially focused on combining kernel approximation with deep learning architecture. Deep Fried Convnets is a deep learning architecture that replaces dense layers of a convolutional neural architecture by a Fastfood approximation of a kernel. This allows to take advantage of the low complexity cost in terms of computation and memory of Fastfood to reduce significantly the computation cost and the number of parameters of the fully-connected layers of the deep convolutional neural network. More formally, let conv(x) be the representation of the data sample x learned by a convolutional neural network. It may include a number of convolution blocks, each including convolution and pooling layers, batch normalization and nonlinear activation. In Deep Fried Convnets, an input x ∈ R d is mapped to the representation spaces conv(x) ∈ R p and then the Fastfood feature map φ f f is applied to the convolutional representation conv(x) instead of the fully-connected layers. The feature representation of x with Deep Fried Convnets is then (φ f f •conv)(x) ∈ R q. It is of note that this method is dedicated to RBF kernels. In, wo architectures have been proposed. The first one relies on the Fastfood kernel approximation method as described above. The second one is a variant of Fastfood called Adaptive-Fastfood. It involves learning the weights of matrices S, G and B through gradient descent rather than setting them randomly, while matrices Π and H are kept unchanged. In the next section we introduce Deepström Networks as an alternative to Deep Fried Convnets. They are based on Nyström approximation and are not limited to RBF kernels. They also allow the use of multiple different kernels and find an appropriate kernel function.3 Deepström NETWORKS In this section, we describe our new Deepström model which combines the desirable characteristics of Nyström approximation and convolutional neural networks. First, we start by revisiting the concept of Nyström kernel approximation from a feature map perspective. Nyström approximation from an empirical kernel map perspective The empirical kernel map is an explicit n-dimensional feature map that is obtained by applying the kernel function on the training data x i. It is defined as DISPLAYFORM9 An interesting feature of the empirical kernel map is that if we consider the inner product in R n ·, · M = ·, M · with a positive semi-definite (psd) matrix M, we can recover the kernel matrix K using the empirical kernel map by setting M equals to the inverse of the kernel matrix. In other words, DISPLAYFORM10 = K. Since K is a psd matrix, one can consider the feature φ emp: DISPLAYFORM11 as an explicit feature map that allows to reconstruct the DISPLAYFORM12 The Deepström network architecture involves a usual convolutional part, conv, including multiple convolutional blocks, and a Deepström layer which is then fed to (eventually multiple) standard dense layers up to the classification layer. The Deepström layer computes the kernel between the output of the conv block for a given input and the corrsponding representations of the trains samples in the subsample L.kernel matrix. This feature map is of dimension n and then is not interesting when the number of example is large. The feature map of the Nyström approximation is given by DISPLAYFORM13 T with x i ∈ L. From an empirical kernel map point of view,φ nys (x) can be seen as an "empirical kernel map" and K DISPLAYFORM14 11 as a metric in the "empirical feature space". From this viewpoint, we think that it could be useful to learn a metric W in the empirical feature space instead of assuming it to be equal to K 11. In a sense, this should allow to learn a kernel by learning its Nyström feature representation. In the following, we call the setting where W is learned by the network Adapative Deepström Network. Principle Deepström networks we propose are an alternative to Deep Fried Convnets. They are based on using the Nyström approximation rather than the Fastfood one to integrate any kernel function on top of convolutional layers of a deep net. Indeed, although Deep Fried Convnets yield state-of-the-art with a significant gain with respect to memory resource and to inference complexity, it is restricted to the Gaussian kernel Fastfood, which may not be always the best choice in practice. In addition, our method can deal with multiple kernels. Deepström nets are Neural Networks that make use of a nonlinear representation function computed with the Nyström approximation (see Figure 1). Starting from Deep Fried Convnets we replace φ f f with φ nys so that a Deepström net implements a function f (x) = (lc • φ nys • conv)(x). In order to compute the above Nyström representation of a sample x one must consider a subsample L of training instances. Since the kernel k is computed on the representations given by convolutional layers, the samples in L must be represented in the same space, and hence must be processed by the convolutional layers as well. Once convolutional representations are calculated, the kernel function may be computed with an input sample and each instance in L in order to get the k x,L, which is then linearly transformed by W before the linear classification layer (see Figure 1).Two main structural differences between Deep Fried Convnets and our Deepström nets: (I) Nyström has the flexibility to use different kernel functions and to combine multiple kernels, and (II) in contrast to Fastfood the Nyström approximation is data dependent. However, one problem arises with the computation of K We present a series of experimental that explore the potential of Deepström networks with respect to various classification settings. First we consider a rather standard setting and compare our approach with standard models on image classification tasks. We explore in particular the behaviour of Deepström networks with various kernels and stress the very limited subsample size needed to reach state-of-the-art accuracy. Next we investigate the use of Deepström networks in a small training set setting, which shows that our approach may allow to learn new classes with only very few training samples, taking advantage of the reduced number of parameters learned by our model. Before describing all these we detail the datasets used. Finally we investigate first the multiple kernel architecture and illustrate its interest when learning with RBF kernel to overcome the hyperparameter selection, and second we demonstrate the benefit of a multiple Deepström approach, combining kernels computed from individual feature maps. We conducted experiments on four well known image classification datasets: MNIST, SVHN, CIFAR10 and CIFAR100 BID7, details on these datasets are provided in Table 1. We pretrained the convolutional layers using standard architectures on both datasets: Lenet for MNIST and VGG19 BID18 for SVHN, CIFAR10 and CIFAR100. We slightly modified the filters' sizes in Lenet network to ensure that the dimension of data after the convolution blocks is a power of 2 (needed for the Deep Fried Convnets architecture).We compare three convolutional architectures in all conducted experiments. Pretrained convolutional parts are shared by the three architectures, which differ from the layers on top of it: Dense architectures use dense hidden layers, i.e. these are classical convnets architectures; Deep Fried implements the Fastfood approximation (Equation 3); Deepstrom stands for our proposal. For Dense architectures, we considered one hidden layer with relu activation function, and varied the output dimension as {2, 4, 8, 16, 32, 64, 128, 1024} in order to highlight accuracies as a function of the number of parameters. For the Fastfood approximation in Deep Fried Convnets we consider that φ f f is gained with one stack of random features to form V in equation 3, except in the experiments of section 4.3 which yields a representation dimension up to 5 times larger. Regarding our approach φ nys, we varied the subset size L ∈ {2, 4, 8, 16, 32, 64, 128}, we tested with the linear, the RBF, and the Chi2 kernels, and we chose as output dimension the same size as the subset sample size. Finally we explored the adaptive as well as non-adaptive variants. Models were learned to optimize the cross entropy criterion with Adam optimizer and a gradient step fixed to 1e −4. Dropout was used on representation layers with probability equal to 0.5. By default the RBF bandwidth was set to the inverse of the mean distance between the representations, after the convolutional part, of pairs of training samples. All experiments were performed with Keras and Tensorflow. Note that the aim of all the experiments below is to investigate the potential of out architecture, not to reach or beat state of the art on the datasets considered. We then compare gained with our architecture and with state-of-the-art models, given a shared convolutional model. Consequently, we did not use tricks such as data augmentation and extensive tuning and, in particular, we did not use the best known convolutional architecture for each of the dataset, we rather used a reasonable deep architecture, VGG19, for the three datasets CIFAR10, CIFAR100 and SVHN. We compare now Deepström networks to two similar architectures, Deep Fried Convnets and classical convolutional networks (inspired from VGG19 and Lenet depending on the dataset). We vary the number of parameters of each architecture in order to highlight classification accuracy with respect to needed memory space. FIG2 shows the compared networks accuracy with respect to the number of parameters, and ignore parameters for convolutions layers to ease the readability. We repeated each experiments 10 times and plot average scores with standard deviations. Deepström models of increasing complexity (number of parameters) correspond to the use of subsample of increasing size from 2 (leftmost point) to 128 (rightmost point). One may see that there is no need of a large subsample here. This may be explained since the convolutional part of the network has been learned to yield quite robust and stable representations of input images. We provide a figure in the Appendix that illustrates this. The Deepström network is able to reach state-of-the-art performance using much fewer parameters than both classical networks and Deep Fried Convnets. Moreover, we also observe smaller variations that points out the robustness of our model. The flexibility in the choice of the kernel function is a clear advantage of out method. The best kernel is clearly dependent on the dataset (linear on MNIST, Chi2 on SVHN and CIFAR100, RBF on CIFAR10). While Random Features in DeepFried are restricted to RBF kernels, we show for instance a gain by using the Chi2 Kernel (k(x 1, x 2) = ||x 1 −x 2 || 2 /(x 1 +x 2)) that had been used for image classification BID20. We also notice the benefit of adaptive variants of Deepström model, suggesting that our model is able to learn and adapt useful Kernel function. Finally, note that we obtained very similar with neural architectures exploiting two hidden layers instead of one after the convolution module conv. Here we explore the ability of our model to work with few training samples, from very few to tens of samples per class. It is an expected benefit of the method since the use of kernels could take advantage of small training samples. Note that we do not exactly deal with a real small training set setting. These preliminary experiments aim to show how the final layers of a convolutional model may be learned from very few samples, given a frozen convolutional model. We actually performed the following experiments by exploiting a trained convolution model conv that has been learned on the full CIFAR100 training set and investigate the performance of Deepström architectures as a function of the training set used to learn the classification layers. One perspective of this work is to exploit such a strategy for domain adaptation settings where the convolutional model is trained on a training set within a different domain as the classes to be recognized. Having at our disposal such a trained convolution model conv, we leverage on the additional information that one may easily include in our models, which is brought by the subsample set. Notice that this subsample may include unlabeled samples since their labels are not used for optimizing the model. Table 2 reports the comparison of network architectures on four datasets. We consider Adaptive Deepström using Linear, RBF or Chi2 kernels and compare with Dense and Adaptive Deepfried for training set sizes of 5, 10 and 20 samples per class. We only consider here adaptive variants since they brought better than their non adaptive counterparts. We obtain models with different complexities: by increasing the hidden layer size in standard convolutional models, or by stacking the number of matrices V in DeepFried (up to 8 times, more was untractable on our machines), and by increasing the subset size in Deepström. Reported are averaged over 30 runs. One may see first that Deepstrom architectures outperfom baselines on every setting except for 5 training samples per class on MNIST. The linear kernel performs well on MNIST but is significantly worse than baselines on harder datasets. At the opposite, both ADSR and ADSC significantly outperfom Adaptive DeepFried for any dataset and perform on par or significantly better than Dense architectures on the hardest CIFAR100 dataset. Moreover one sees that no single kernel based Deepstrom architecture dominate on all settings, showing the potential interest of combining multiple kernels as following experiments will show. We report here gained using multiple kernels in two different ways. First we exploited the Multiple Kernels strategy that we described in section 3 for exploiting multiple kernels in the output of the convolutional blocks, conv. FIG4 reports gained when using a combination of RBF kernels with various bandwidths and for different subsample sizes. Our multiple kernel strategy, exploiting kernels defined with various values of the hyperparameter allows automatically handling this hyper-parameter which usually requires to be tuned either through cross validation or to be manually chosen. The plots show the accuracy on the CIFAR10 dataset as a function of σ value, where the performance of the multiple kernel Deepström is shown as a horizontal line. Plots report for various subsample size equal to 2 (left), 4 (middle) and 8 (right), averaged over 10 runs. As may be seen, using our Multiple kernel strategy allows adapting the kernel combination optimally from the data without requiring any prior choice on the RBF bandwith hyper-parameter. Second, we investigated another architecture that exploits Multiple Deepström approximations as presented in section 3. Here we use in parallel multiple Nyström approximations where kernels are dedicated to deal each with the output of a single feature map of the conv part. TAB4 reports on CIFAR100. We show the best performances obtained for each method by grid-searching on various hyper-parameters depending on the models, within a similar range of number of parameters. For Dense model, we considered one or two hidden layers of 16, 64, 128, 1024, 2048 or 4096 neurons. Deepfried is the adaptive variant where we varied the number of stacks in 1, 3, 5, 7. Deepström is also the adaptive variant where the subsample size is in 16, 64, 128, 256, 512. We observe that both Deepström models outperform the considered baselines, demonstrating the interest in combining Deepström approximations. We proposed Deepström, a new hybrid architecture that mixes deep networks and kernel methods. It is based on the Nyström approximation that allow considering any kind of kernel function in contrast to Deep Fried Convnets. Our proposal allows reaching state of the art while significantly reducing the number of parameters on various datasets, enabling in particular learning from few samples. Moreover the method allows to easily deal with multiple kernels and with multiple Deepström architectures. FIG5 plots the 2-dimensional φ nys representations of some CIFAR10 test samples obtained with a subsample of size equal to 2 (while the number of classes is 10) and two different kernels. One may see here that the 10 classes are already significantly well separated in this low dimensional representation space, illustrating that a very small sized subsammple is already powerfull. Beside, we experienced that designing Deepström Convnets on lower level features output by lower level convolution blocks may yield state-of-the-art performance as well while requiring larger subsamples."

We propose a new method for using ontology information to improve performance on massively multi-label prediction/classification problems.

Represent the short summary for finding the corresponding scientific article (long)


Question: Generative Adversarial Networks have made data generation possible in various use cases, but in case of complex, high-dimensional distributions it can be difficult to train them, because of convergence problems and the appearance of mode collapse. Sliced Wasserstein GANs and especially the application of the Max-Sliced Wasserstein distance made it possible to approximate Wasserstein distance during training in an efficient and stable way and helped ease convergence problems of these architectures. This method transforms sample assignment and distance calculation into sorting the one-dimensional projection of the samples, which a sufficient approximation of the high-dimensional Wasserstein distance. In this paper we will demonstrate that the approximation of the Wasserstein distance by sorting the samples is not always the optimal approach and the greedy assignment of the real and fake samples can faster convergence and better approximation of the original distribution. Generative Adversarial Networks (GANs) were first introduced in , where instead of the application of a mathematically well-established loss function an other differentiable neural network, a discriminator was applied to approximate the distance between two distributions. These methods are popularly applied in data generation and has significantly improved the modelling capabilities of neural networks. It was demonstrated in various use cases that these approaches can approximate complex high-dimensional distributions in practice , ,. Apart from the theoretical advantage of GANs and applying a discriminator network instead of a distance metric (e.g. 1 or 2 loss), modelling high-dimensional distributions with GANs often proves to be problematic in practice. The two most common problems are mode collapse, where the generator gets stuck in a state where only a small portion of the whole distribution is modeled and convergence problems, where either the generator or the discriminator solves his task almost perfectly, providing low or no gradients for training for the other network. Convergence problems were improved, by introducing the Wasserstein distance , which instead of a point-wise distance calculation (e.g. cross-entropy or 1 distance) calculates a minimal transportation distance (earth mover's distance) between the two distributions. The approximation and calculation of the Wasserstein distance is complex and difficult in highdimensions, since in case of a large sample size calculation and minimization of the transport becomes exponentially complex, also distance can have various magnitudes in the different dimensions. it was demonstrated that high-dimensional distributions can be approximated by using a high number of one dimensional projections. For a selected projection the minimal transport between the one dimensional samples can be calculated by sorting both the real and the fake samples and assigning them according to their sorted indices correspondingly. As an additional advantage, it was also demonstrated in that instead of the regular mini-max game of adversarial training, the distribution of the real samples could be approximated directly by the generator only, omitting the discriminator and turning training into a simple and more stable minimization problem. The theory of this novel method is well described and it was demonstrated that it works in practice, but unfortunately for complex, high-dimensional distributions a large amount of projections are needed. it was demonstrated how the high number of random projections could be substituted by a single continuously optimized plane. The parameters of this projection are optimized in an adversarial manner selecting the "worst" projection, which maximizes separation between the real and fake samples using a surrogate function. This modification brought the regular adversarial training back and created a mini-max game again, where the generator creates samples which resemble well to the original distribution according to the selected plane and the discriminator tries to find a projection, which separates the real and fake samples from each other. The essence of Sliced Wasserstein distances is how they provide a method to calculate minimal transportation between the projected samples in one-dimension with ease, which approximates the Wasserstein distance in the original high-dimension. In theory this approach is sound and works well in practise. It was proven in that the sliced Wasserstein distance satisfies the properties of non-negativity, identity of indiscernibles, symmetry, and triangle inequality, this way forming a true metric. However it approximates high-dimensional distributions well, we would like to demonstrate in this paper that the assignment of real and fake samples by sorting them in one dimension also has its flaws and a greedy assignment approach can perform better on commonly applied datasets. We would also argue regarding the application of the Wasserstein distance. We will demonstrate that in many cases various assignments can the same minimal transportation during training and calculation of the Wasserstein distance with sorting can alter the distribution of perfectly modeled samples even when only a single sample differs from the approximated distribution. Generative adversarial networks can be described by a generator (G), whose task is to generate a fake distribution (P F), which resembles a distribution of real samples (P R) and a discriminator, whose task is to distinguish between P F and P R samples. Formally the following min-max objective is iteratively optimized: min Where V is a distance measure or divergence between the two distributions. the Wasserstein-p distance was proposed to improve the staibility of GANs, which can be defined as: where p is a parameter of the distance, and (P F, P R) defines all the possible joint distributions over P F and P R. The number of possible joint distributions increases factorially with the number of samples and the calculation of the minimal transport can be difficult. The instability and high complexity of wasserstein GANs were further improved by the introduction of the sliced Wassersstein distance in , which can be defined as: where P w F, P w R are one-dimensional projections of the high dimensional real and fake samples to w and Ω denotes a sufficiently high number of projections on the unit sphere. In this setting the Wasserstein distance can be calculated by sorting both the real and fake samples and assigning them by their indices, instead of checking all the possible joint distributions in (P w F, P w R). Max-sliced Wasserstein GANs can be introduced by selecting the worst projected distribution from Ω noted by w max and since the projection can be implemented by a fully connected layer, one can re-introduce the mini-max game used in regular GAN training: min In this setup the distance in the single, one-dimensional projected space can be calculated by sorting the samples in the similar manner as in case of the sliced Wasserstein distance. In case of high sample number and complex high-dimensional distributions the optimal transport can often be calculated using various assignments between two sets of samples. For example if a sample containing n elements from P F can be defined by the series of F w 1...n after the projection, and a sample from P R is defined by R w 1...n and we know that for every i and j (i, j ∈ 1...n): Which means that the projected samples are well separated from each other (all the projected fake samples are smaller than any of the projected real samples), for p = 1 all possible assignments of the i, j pairs will the same distance, which is the minimal transportation of the samples, although the pairwise differences can be very different. One can also easily see, that the minimal transport calculated with sorting might not assign identical samples to each other, which is depicted on Fig. 1. As it can be seen from the figure, for example in case of two Gaussian distributions with the same variance, but different mean values using sorting to calculate the minimal transport (along arbitrary dimensions), will the shift of one of the distributions (in this case green). This transformation will also affect those samples which are at the intersection of the two distributions. This means that there might be identical pairs in (P F and P R) generating an error, which is not correct. One could assume that if a generator produces a sample identical to one of the samples of P R it should not generate an error and no gradients should be invoked by this sample pair. In this case the assignment will not pair these identical samples for comparison. Figure 1: This figure depicts a flaw of the Wasserstein distance. In these figures a blue (desired output distribution and green, generated fake distribution can be seen), the assignments/desired transports are illustrated by red arrows. Wasserstein distance calculates minimal transportation between all the samples. In many cases this shifting all the samples to cover the whole distribution correctly, this is depicted on the left subfigure. Unfortunately this means that those samples which are at the intersection of the two distributions, meaning that for these generated samples an identical pair could be found in the real samples, will also be altered and shifted towards an other part of the real distribution. Instead of this we propose the calculation of the assignment using a greedy method. This will ensure that that identical (or similar) sample pairs will be selected first, and after this the transportation between the disjunct regions will be calculated, which is depicted on the right subfigure. Instead of sorting the samples we propose a method which assigns similar samples to each other. First we would like to remove the intersection of the two distributions and calculate the minimal transport for the remaining disjunct regions. By this calculation the value of the minimal transport will be the same (in case of a sufficiently large sample number), but more similar samples will be assigned to each other. The previous example reveals a caveat of the Wasserstein distance, that it optimizes the global transport between two distributions, but ignores identical sample pairs or intersecting regions of density functions in them. Unfortunately identical samples will be extremely rarely found in two distributions, but closer sample pairs should also be assigned to each other. In training one typically uses mini-batches and will not see the whole distribution, certain parts or elements might not be present in each mini-batch if they probability in the original distribution is lower than one over the minibatch size. This can cause further problems in the calculation of the minimal transport using sorting, since the appearance of one wrongly generated sample at the first or last position of the projected samples can a completely different assignment. This problem is depicted on Fig. 2. We have to emphasize that although the assignment and distance calculation happen in one dimension, these distances are of the projection of high dimensional embeddings, where there can exist dimensions which can decrease the distance between the two samples without significantly changing the projected position of all the other samples. Figure 2: In this figure we would like to demonstrate different approaches for minimal transportation in one-dimension using mini-batches. Real samples are noted by blue and fake samples are plotted by red stars. The position of the samples were shifted vertically for better display, but only their horizontal position matters. On both figures the same constellation can be seen: four similar sample pairs and one which is very different. The left subfigure depicts sample assignment by sorting, meanwhile the right figure depicts assignment by a greedy approach. The summed distances are the same in both cases for p = 1. We would like to argue that the lower assignment can be better for network training. We introduce greedy training of max sliced Wasserstein GANs, where instead of sorting the samples of the projections we iteratively select the most similar pairs of them for loss calculation. Our training algorithm as a pseudocode is defined in Algorithm 1. The training process is really similar to the approach introduced in. The main and only alteration is between line ten and sixteen, where instead of the sorting operation of the original approach we first generate a matrix which determines the distances for all possible sample pairs in one-dimension. First we select the smallest element in this matrix and remove its row and column and iterate this process until the matrix contains only one element which will be the distance between the last two, least similar sample pairs. We have to note that our approach requires O(n 3) steps compared to the original O(nlog(n)) operations of sorting. We have to implement n minimum selections in the distance matrix with size n × n. We also have to note that this increase in complexity refers for training only and has no effect ton the inference complexity of the generator, also n is the sample or mini-batch size during training, which is usually a relatively low number. In our experiences, using batches of 16 to 512 samples, this step did not a significant increase in computation time during training. In the previous section flaws of the Wasserstein distance using sorting in one-dimension were presented and greedy sample assignment was suggested as a possible method fixing these defects. On the other hand one has to admit that in certain cases greedy approach does not ensure minimal transportation, transportation ed by sorting will always be lower or equal to the assignment calculated by the greedy approach. We also have to state that distance calculation with the greedy assignment does not form a proper metric since it does not satisfy the triangle inequality. It can also easily be seen that one could generate one-dimensional cases where the greedy assignment is arbitrarily larger than sorting. To ensure that these cases can not occur during training we have introduced a hybrid approach where we first sort the samples and calculate their Wasserstein distance (noted as W S P) and after this step we also calculate the distance using greedy assignment (noted as Algorithm 1 Training the Greedy Max-Sliced Wasserstein Generator Given: Generator parameters θ g, discriminator parameters θ d, ω d, sample size n, learning rate α 1: while θ g not converged do 2: Sample data D i n i=1 compute surrogate loss s(ωD i, ωF end for 8: Sample data D i n i=1 Create distance matrix: for k = 1 → n do 13: Find min value of M : L ← L + m s,t Remove Row s and Column t from M end for 17: . In this case a parameter (ν) can be set, determining a limit and if the difference between the two distances is larger than this value the sorted distance will be used. This way Wasserstein distance with the hybrid assignment (W H P) can be defined as: In case of ν = 0 the greedy assignment will only be used in those cases, where it also the minimal transportation. If ν is set to one, the greedy assignment will be used if the distance calculated this way is less than twice the minimal distance. We were using ν = 1 in our experiments. It is also worth to note, that sorting the samples before creating the distance matrix (M) can help finding the minimal element, since in this case all series starting from the minimal element in a selected row or column will form a monotonically increasing series. For the first tests we have generated a one-dimensional toy problem using Gaussian Mixture Model with five modes, each of them with variance of 0.15 and expected values of 0, 2, 4, 6 and 8 accordingly. We have trained a four layered fully connected network containing 128, 256, 512 and 1 neurons in the corresponding layers for 5000 iterations using Adam optimizer, with learning rate of 0.001, with batches of 64. No discriminator was applied in this case, the position of the generated samples were optimized directly by the generator and distance calculation was done directly on the output of the generator. In one setup the loss was calculated by sorting the samples, in the other setup we used the greedy approach introduced in Algorithm 1. For further details regarding the hyperparameters of all simulations and for the sake of reproducibility our code can be found on github, containing all details of the described simulations: the link was removed from the text but for the sake of anonymity, but a link pointing to the codes was used during submission After training we have used the generator to generate 16000 samples and we have compared it to 16000 samples generated by the Gaussian Mixtures Model. We have calculated the KullbackLeibler divergence and the Pearson correlation coefficient between these two distributions, repeated all experiments ten times and averaged the . The can be seen in Table 1. As it can be seen training the network with greedy assignment ed lower Kullback-Leibler divergence and higher correlation between the generated and the original distributions, which signals the the greedy approach could approximate the original distribution of the samples better. For the calculation of the Kullback-Leibler divergence histogram calculation was done using 100 bins uniformly distributed between -0.5 and 8.5. An image plotting the histograms using these parameters can be seen on Fig. 3. Table 1: KL divergence, and Pearson correlation between the original and generated distribution using sorting and a greedy approach. Method KL DIV Pearson Corr Sorted 0.91 0.41 Greedy 0.68 0.74 Figure 3: Histograms of the one-dimensional distributions. The blue curve plots the original distribution of the Gaussian Mixture Model, the green curve depicts the generated distribution using sorting for sample assignment, meanwhile the red curve display the of the greedy approach. We have repeated the simulations described in section 3.1 using two-dimensional Gaussian Mixture models. Nine modes were generated forming a small 3 × 3 grid, each with variance of 0.1 and the distance between neighbouring modes was 3.5 in the grid. In this case a five-layered network with 128, 256, 512, 1024 and 1 neurons was applied and training was executed for 500.000 iterations. All other parameters of the simulation were kept the same. The Kullback-Leibler divergence and Pearson correlation coefficients of these simulations comparing the greedy and sorting approaches can be seen in Table 2, meanwhile random samples are depicted on Fig.??. for qulaitative comparison. In this experiment a two-dimensional histogram was used for the Kullback-Leibler divergence calculation, where a uniform grid was formed between -2 and 8 forming 100 bins. We have evaluated our approach on the MNIST dataset as well, where we have used the DCGAN architecture for image generation. Images were resized to 64 × 64 to match the input dimension of the architecture, but single channel images were used. We were using batches of 128 and 16 samples and Adam optimizer for training both the generator and the discriminator (which was a single projection). Since the comparison of high-dimensional (in this case 4096) distributions is complex, we have binarized the images (thresholded at half of the maximum intensity) and calculated the KL divergence between the distribution of white and black values at each pixel for the original distribution and 60000 generated fake samples. The can be seen in Table 3, meanwhile a qualitative comparison of randomly selected samples can be seen on Figure displays randomly selected samples generated with the same network architecture and training parameters on the MNIST dataset using batches of 16. The samples on the left were generated using sorting assignment with Max-sliced Wasserstein distance, the samples in the middle were generated using greedy sample assignment and the samples on the right were generated using the hybrid approach. All samples were generated after 20 epochs of training. We have also conducted experiments on the resized CelebA dataset , in our case images were downscaled to 64 × 64. We have used the DCGAN architecture and compared three different approaches for sample assignment for distance calculation. We did not use any special normalization or initialization methods during training. We used batches of 16 for training. We have selected 10.000 samples randomly from the dataset and generated 10.000 random projection and used sliced Wasserstein distance with sorting to compare the distributions along these lines. The can be seen in Table 4, meanwhile a qualitative comparison of randomly selected samples can be seen on generated with the same network architecture and training parameters on the CelebA dataset using batches of 16. All samples were generated using Max-sliced Wasserstein distance, the samples on the left were generated using sorting for sample assignment, in the middle the greedy approach was used, meanwhile the samples on the right were generated using the hybrid approach. All samples were generated after 20 epochs of training. In this paper we have introduced greedy sample assignment for Max-Sliced Wasserstein GANs. We have shown that using one-dimensional samples, in many cases multiple assignments can optimal transportation and in most cases sorting changes all the samples, meanwhile those parts of the distribution which are at a "good" position should not generate error. We proposed greedy assignment as a possible solution, where samples will be assigned to their most similar counterparts. We have also introduced how the combination of the two methods can be applied ing a hybrid approach in which it can automatically selected -based on the difference of the two measures -which assignment will be used. We have demonstrated on simple toy datasets that greedy assignment performs better than sorting the samples and we have evaluated both the greedy and the hybrid methods on commonly investigated datasets (MNIST and CelebA). With all datasets the greedy assignment ed lower KullbackLeibler divergence and higher correlation than the traditional approach. We have used the Max-Sliced Wasserstein distance for the base of our comparison, since this is the most recent version, which also the best performance, but all the approaches can be exploited in case of regular Sliced Wasserstein distances as well. Also our approach changes the distance calculation only and it can be applied together with various other improved techniques and architectures which are used in GAN training.

Represent this text to find a much much longer article about it with say hundreds-thousands of words!

We apply a greedy assignment on the projected samples instead of sorting to approximate Wasserstein distance

represent the tldr for finding the corresponding scientific article

Our proposed algorithm does not use all of the unlabeled data for the training, and it rather uses them selectively.